Sep 16 18:44:56.892170 ip-10-0-133-194 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:57.318031 ip-10-0-133-194 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:57.318031 ip-10-0-133-194 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:57.318031 ip-10-0-133-194 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:57.318031 ip-10-0-133-194 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:57.318031 ip-10-0-133-194 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:57.318686 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.318134    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:57.322161 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322144    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:57.322161 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322160    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322164    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322168    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322172    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322175    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322178    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322181    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322184    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322186    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322189    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322191    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322194    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322197    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322199    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322202    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322205    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322207    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322210    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322212    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322215    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:57.322228 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322218    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322220    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322223    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322226    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322229    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322231    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322234    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322237    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322240    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322243    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322246    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322248    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322252    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322256    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322259    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322262    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322265    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322268    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322271    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:57.322783 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322275    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322278    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322281    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322284    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322288    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322290    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322294    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322299    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322301    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322304    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322307    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322310    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322313    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322315    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322318    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322321    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322324    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322328    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322331    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:57.323291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322333    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322336    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322338    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322341    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322343    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322346    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322348    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322351    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322353    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322356    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322359    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322362    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322365    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322367    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322370    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322373    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322376    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322378    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322381    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322384    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:57.323761 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322387    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322389    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322392    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322395    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322397    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322400    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322402    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322405    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322408    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322411    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322414    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322417    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322419    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322422    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322425    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322427    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322430    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322432    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322435    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.322437    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:57.324259 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323132    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323139    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323142    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323145    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323148    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323151    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323154    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323157    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323160    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323163    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323165    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323168    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323170    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323173    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323175    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323178    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323181    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323183    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323185    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323188    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:57.324731 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323193    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323196    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323199    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323201    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323204    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323207    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323209    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323212    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323214    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323217    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323219    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323222    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323224    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323227    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323229    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323232    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323234    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323237    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323240    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:57.325236 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323244    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323249    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323252    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323255    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323258    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323261    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323264    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323267    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323270    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323272    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323275    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323278    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323280    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323283    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323286    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323289    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323291    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323294    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323296    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:57.325706 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323300    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323304    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323307    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323309    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323312    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323315    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323317    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323319    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323322    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323324    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323328    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323330    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323333    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323336    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323338    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323341    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323343    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323346    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323349    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323351    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:57.326172 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323354    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323356    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323359    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323361    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323364    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323366    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323370    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323373    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323375    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323378    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323380    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323383    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323385    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323388    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323392    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323395    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323398    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323401    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323404    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323406    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:57.326677 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323409    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323764    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323769    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323772    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323774    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323777    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323782    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323785    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323789    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323792    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323795    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323799    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323802    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323805    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323808    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323811    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323814    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323816    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323819    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:57.327242 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323822    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323825    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323827    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323830    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323832    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323835    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323838    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323840    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323843    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323845    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323848    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323850    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323854    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323870    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323873    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323875    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323878    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323880    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323883    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:57.327785 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323886    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323891    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323894    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323896    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323899    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323902    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323904    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323907    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323910    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323912    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323915    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323918    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323920    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323923    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323925    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323928    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323930    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323933    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323935    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323938    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:57.328338 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323941    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323943    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323946    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323948    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323951    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323953    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323956    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323958    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323962    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323964    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323966    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323969    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323971    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323976    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323979    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323981    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323984    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323986    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323989    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323991    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:57.328890 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323994    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.323997    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324016    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324021    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324024    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324027    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324029    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324032    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324034    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324037    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324039    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324042    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324044    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324047    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324049    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324052    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324054    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324057    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324059    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:57.329479 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324062    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324065    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324069    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324150    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324186    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324193    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324198    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324203    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324206    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324210    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324214    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324218    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324221    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324224    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324228    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324231    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324234    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324236    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324240    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324243    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324246    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324249    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324252    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:57.329999 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324255    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324258    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324262    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324266    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324269    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324272    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324275    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324278    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324281    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324284    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324287    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324290    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324296    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324299    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324302    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324306    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324309    2581 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324313    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324318    2581 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324321    2581 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324324    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324327    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324330    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324333    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:57.330620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324336    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324339    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324342    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324345    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324348    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324351    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324354    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324357    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324360    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324362    2581 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324366    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324369    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324372    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324375    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324379    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324381    2581 flags.go:64] FLAG: --help="false"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324384    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324387    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324390    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324393    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324398    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324401    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324404    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:57.331320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324408    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324411    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324414    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324417    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324420    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324423    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324426    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324429    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324432    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324435    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324438    2581 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324441    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324444    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324447    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324452    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324455    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324458    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324460    2581 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324463    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324467    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324470    2581 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324472    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324476    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324479    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324483    2581 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:57.331910 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324486    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324489    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324492    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324495    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324499    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324502    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324505    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324514    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324517    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324520    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324523    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324526    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324532    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324535    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324538    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324541    2581 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324544    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324547    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d8f3656a7dec7f20"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324550    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324553    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324556    2581 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324559    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324562    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324565    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:57.332529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324568    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324571    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324574    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324577    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324580    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324583    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324586    2581 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324589    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324592    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324595    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324598    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324601    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324604    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324608    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324611    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324615    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324618    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324621    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324624    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324627    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324631    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324633    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324639    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324641    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324644    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:57.333112 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324648    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324651    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324654    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324657    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324660    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324663    2581 flags.go:64] FLAG: --v="2"
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324666    2581 flags.go:64] FLAG: --version="false"
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324670    2581 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324674    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.324677    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324786    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324790    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324793    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324796    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324799    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324802    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324805    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324808    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324810    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324813    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324818    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324821    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:57.333738 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324825    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324827    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324830    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324833    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324835    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324838    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324842    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324844    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324847    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324850    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324853    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324868    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324871    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324873    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324876    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324879    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324881    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324884    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324887    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324890    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:57.334293 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324892    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324895    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324897    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324900    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324903    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324905    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324908    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324910    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324913    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324915    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324919    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324922    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324925    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324927    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324931    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324933    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324936    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324939    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324942    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324945    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:57.334817 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324948    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324951    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324953    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324956    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324958    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324961    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324963    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324966    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324968    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324971    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324974    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324976    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324979    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324981    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324985    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324989    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324992    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324995    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.324999    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:57.335318 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325002    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325005    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325008    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325012    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325014    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325017    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325020    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325022    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325025    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325028    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325030    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325033    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325036    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325039    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325041    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325044    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325046    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325049    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325052    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:57.335819 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325056    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325059    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325062    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325065    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325068    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325070    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325073    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325076    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.325078    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.325086    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.332381    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.332396    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.332415    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.332425    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:57.336320 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332479    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332484    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332488    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332491    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332494    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332497    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332499    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332502    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332505    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332509    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332511    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332514    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332517    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332520    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332522    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332525    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332527    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332530    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332533    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332536    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:57.336707 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332539    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332541    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332544    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332547    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332550    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332553    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332556    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332558    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332561    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332565    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332569    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332571    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332574    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332577    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332580    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332582    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332585    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332588    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332590    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:57.337310 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332593    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332597    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332601    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332604    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332607    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332610    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332613    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332616    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332618    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332621    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332624    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332627    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332630    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332633    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332635    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332638    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332641    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332643    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332646    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332649    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:57.337942 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332651    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332654    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332657    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332661    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332663    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332666    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332669    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332672    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332674    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332677    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332679    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332682    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332686    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332689    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332691    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332694    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332697    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332699    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332702    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:57.338569 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332706    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332708    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332711    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332713    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332716    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332718    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332721    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332724    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332726    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332729    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332732    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332734    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332737    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332740    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332742    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332745    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332749    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332753    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332756    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:57.339201 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332759    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332762    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.332767    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332905    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332911    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332914    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332918    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332920    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332923    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332926    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332929    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332932    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332935    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332938    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332940    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:57.339776 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332943    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332946    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332948    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332951    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332954    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332957    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332961    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332964    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332967    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332970    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332973    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332975    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332978    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332980    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332983    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332986    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332989    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332992    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332994    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332997    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:57.340183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.332999    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333002    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333005    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333007    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333010    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333012    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333015    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333018    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333020    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333022    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333026    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333028    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333031    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333033    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333036    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333038    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333041    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333044    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333046    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333049    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:57.340670 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333051    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333054    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333056    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333059    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333061    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333064    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333066    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333069    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333071    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333074    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333077    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333079    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333082    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333084    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333087    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333089    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333092    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333095    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333097    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333099    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:57.341283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333102    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333105    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333107    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333110    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333112    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333115    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333118    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333120    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333123    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333126    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333128    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333131    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333135    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333138    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333141    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333144    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333146    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333149    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333152    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:57.341758 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333154    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333157    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333159    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333164    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333167    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333169    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333172    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:44:57.333174    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.333179    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.333279    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.335561    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.338960    2581 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.339012    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:57.342226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.339039    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:57.342569 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.342391    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:57.343706 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.343688    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:57.354876 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.354844    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:57.358148 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.358132    2581 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:57.359058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.359044    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:57.362384 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.362366    2581 fs.go:135] Filesystem UUIDs: map[3320db51-5407-4ff7-bc76-82fc3026e7e4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d17c05b8-ea5b-462c-ab55-4ad6c3b3702a:/dev/nvme0n1p4]
Sep 16 18:44:57.362452 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.362384    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:57.368542 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.368433    2581 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:57.366950781 +0000 UTC m=+0.371965693 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099960 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253232a469d53735e91953d590cd0a SystemUUID:ec253232-a469-d537-35e9-1953d590cd0a BootID:37fb030e-aac5-4d63-9f8c-6a9ad886825d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ca:b5:5f:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ca:b5:5f:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:30:d5:39:47:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:57.368542 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.368539    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:57.368651 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.368610    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:57.371190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.371172    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:57.372162 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.372139    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:57.372295 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.372165    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:57.373047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.373036    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:57.373082 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.373050    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:57.373082 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.373058    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:57.373693 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.373684    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:57.374890 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.374882    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:57.379285 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379274    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:57.379407 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379397    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:57.379438 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379428    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:57.379472 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379462    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:57.379500 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379490    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:57.379529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379508    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:57.379529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.379526    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:57.380217 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.380203    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:57.380281 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.380232    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:57.384273 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.384232    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:57.387271 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.387256    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:57.387322 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.387287    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:57.389291 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389280    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389302    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389310    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389318    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389330    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389337    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389353    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389362    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:57.389372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389373    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:57.389580 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389380    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:57.389580 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389389    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:57.389580 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.389394    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:57.390331 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.390319    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:57.390368 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.390332    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:57.390940 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.390922    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:57.390990 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.390921    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:57.394057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.394044    2581 server.go:1268] "Started kubelet"
Sep 16 18:44:57.394211 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.394170    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:57.394272 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.394236    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:57.394319 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.394134    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:57.395015 ip-10-0-133-194 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:57.395731 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.395715    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:57.397502 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.396926    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:57.398456 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.398441    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:57.400701 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.400685    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:57.401963 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.400727    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-194.ec2.internal.18d5e16fd5c99e53  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-194.ec2.internal,UID:ip-10-0-133-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-194.ec2.internal,},FirstTimestamp:2026-09-16 18:44:57.394019923 +0000 UTC m=+0.399034836,LastTimestamp:2026-09-16 18:44:57.394019923 +0000 UTC m=+0.399034836,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-194.ec2.internal,}"
Sep 16 18:44:57.402406 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.402392    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:57.402456 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.402405    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:57.402500 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.402487    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.403156 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403139    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:57.403264 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403166    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:57.403264 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403177    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:57.403376 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403273    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:57.403376 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403282    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:57.403490 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403469    2581 factory.go:55] Registering systemd factory
Sep 16 18:44:57.403490 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403492    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:57.403696 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403686    2581 factory.go:153] Registering CRI-O factory
Sep 16 18:44:57.403696 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403697    2581 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:57.403792 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403749    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:57.403792 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403773    2581 factory.go:103] Registering Raw factory
Sep 16 18:44:57.403792 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.403788    2581 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:57.404216 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.404197    2581 manager.go:349] Starting recovery of all containers
Sep 16 18:44:57.409466 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.409441    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:44:57.409545 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.409520    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:44:57.415138 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.415116    2581 manager.go:354] Recovery completed
Sep 16 18:44:57.415949 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.415776    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:57.418005 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.417983    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hjs55"
Sep 16 18:44:57.421650 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.421635    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.423214 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423197    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.423291 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423224    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.423291 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423234    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.423291 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423242    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hjs55"
Sep 16 18:44:57.423749 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423737    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:57.423749 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423749    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:57.423817 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423752    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.423817 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.423763    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:57.424394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.424372    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.424464 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.424403    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.424464 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.424417    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.425220 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.425159    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-194.ec2.internal.18d5e16fd7870f5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-194.ec2.internal,UID:ip-10-0-133-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-194.ec2.internal,},FirstTimestamp:2026-09-16 18:44:57.42321238 +0000 UTC m=+0.428227295,LastTimestamp:2026-09-16 18:44:57.42321238 +0000 UTC m=+0.428227295,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-194.ec2.internal,}"
Sep 16 18:44:57.426515 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.426500    2581 policy_none.go:49] "None policy: Start"
Sep 16 18:44:57.426515 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.426517    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:57.426613 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.426528    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:57.428308 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.428297    2581 policy_none.go:47] "Start"
Sep 16 18:44:57.454261 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.454248    2581 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:57.454358 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.454279    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:57.454358 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.454289    2581 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:57.454606 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.454592    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:57.454650 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.454610    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:57.454967 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.454952    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:57.455043 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.455024    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:57.455076 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.455044    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:57.455602 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.455582    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:57.455690 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.455630    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.514645 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.514616    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.514677    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.514703    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.514798    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.516690    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.555308    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.556118    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.556147    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.556158    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.556180    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.562735    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.564047 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.562754    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-194.ec2.internal\": node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.590728 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.590658    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.615276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.615254    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"]
Sep 16 18:44:57.615348 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.615341    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.616198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.616183    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.616284 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.616213    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.616284 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.616226    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.618474 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.618460    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.618608 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.618593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.618660 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.618622    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.619142 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.619125    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.619226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.619150    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.619226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.619172    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.619226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.619181    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.619226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.619153    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.619370 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.619231    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.621310 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.621294    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.621398 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.621318    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:57.621918 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.621905    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:57.621991 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.621929    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:57.621991 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.621945    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:57.649438 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.649419    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-194.ec2.internal\" not found" node="ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.653813 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.653799    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-194.ec2.internal\" not found" node="ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.691452 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.691432    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.705197 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.705176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.705272 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.705199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.705272 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.705225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bee55f8a1340335ec8a8808d97d355d4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"bee55f8a1340335ec8a8808d97d355d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.792411 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.792372    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.805693 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.805675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bee55f8a1340335ec8a8808d97d355d4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"bee55f8a1340335ec8a8808d97d355d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.805760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.805702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.805760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.805718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.805760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.805744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.805849 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.805778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02f1e051c1877a9e2bbc49221450c3da-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal\" (UID: \"02f1e051c1877a9e2bbc49221450c3da\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.805849 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.805783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bee55f8a1340335ec8a8808d97d355d4-config\") pod \"kube-apiserver-proxy-ip-10-0-133-194.ec2.internal\" (UID: \"bee55f8a1340335ec8a8808d97d355d4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.893182 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.893108    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:57.952558 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.952538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.956220 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:57.956201    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 16 18:44:57.993773 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:57.993749    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:58.094335 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:58.094297    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:59.895837 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:59.895772    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:44:59.925257 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:44:59.925230    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:59.996879 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:44:59.996840    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-194.ec2.internal\" not found"
Sep 16 18:45:00.039383 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.039356    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:45:00.039888 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.039513    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:00.039888 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.039545    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:45:00.054445 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.054402    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:00.081791 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.081760    2581 apiserver.go:52] "Watching apiserver"
Sep 16 18:45:00.093380 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.093358    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:45:00.094890 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.094868    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hxpgk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4","openshift-dns/node-resolver-ptxgf","openshift-multus/multus-additional-cni-plugins-rj4pr","openshift-multus/multus-rnw6q","openshift-multus/network-metrics-daemon-tgznf","openshift-network-diagnostics/network-check-target-qb9wt","kube-system/konnectivity-agent-gxggm","openshift-cluster-node-tuning-operator/tuned-djj5m","openshift-image-registry/node-ca-67d58","openshift-network-operator/iptables-alerter-ttptx","openshift-ovn-kubernetes/ovnkube-node-lwq4x"]
Sep 16 18:45:00.097217 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.097197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.097300 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.097278    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:00.099328 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.099305    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.101549 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.101523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.102812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.102385    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tnt8m\""
Sep 16 18:45:00.102812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.102602    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:45:00.102812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.102679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.103190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.103139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.104137 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.104117    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"
Sep 16 18:45:00.104489 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.104470    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:45:00.104786 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.104766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.105174 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.105152    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qlc8m\""
Sep 16 18:45:00.105325 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.105311    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.105883 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.105851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.108219 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.108313 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108279    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9kn7w\""
Sep 16 18:45:00.108540 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.108627 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:45:00.108690 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108653    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:45:00.108749 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108723    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:45:00.108749 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.108736    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.110511 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.110608 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.110608 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.110547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:00.110608 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-kubelet-config\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmb4f\" (UniqueName: \"kubernetes.io/projected/382ffe97-ecd3-4f98-92bd-47b589bcb233-kube-api-access-fmb4f\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3780632e-fea7-4460-a811-203997052dcb-tmp-dir\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-system-cni-dir\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110757    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9krvl\""
Sep 16 18:45:00.110764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdr8g\" (UniqueName: \"kubernetes.io/projected/3780632e-fea7-4460-a811-203997052dcb-kube-api-access-vdr8g\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-device-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mx2\" (UniqueName: \"kubernetes.io/projected/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-kube-api-access-f9mx2\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3780632e-fea7-4460-a811-203997052dcb-hosts-file\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.110985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-dbus\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.111007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-socket-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.111029    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-registration-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.111073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.111052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-etc-selinux\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.111431 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.111090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-sys-fs\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.111431 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.111119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cnibin\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.112628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.112614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:00.112704 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.112666    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:00.115001 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.114985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.117164 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.117146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.117600 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.117583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:45:00.117691 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.117611    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:45:00.117691 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.117682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xn7fs\""
Sep 16 18:45:00.117939 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.117926    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:45:00.119293 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.119274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.119442 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.119425    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.120027 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.120001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.120123 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.120046    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mp5mh\""
Sep 16 18:45:00.121499 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.121466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.121639 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.121622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.122076 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.122057    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:45:00.122176 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.122113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2l67\""
Sep 16 18:45:00.122176 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.122146    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:00.122263 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.122223    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"
Sep 16 18:45:00.122526 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.122494    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.123670 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.123654    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal"]
Sep 16 18:45:00.123767 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.123756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.123968 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.123946    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.124200 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.124183    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.124487 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.124470    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:45:00.124487 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.124480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r7sbp\""
Sep 16 18:45:00.126600 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.126585    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:45:00.128013 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.127997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:45:00.128108 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.128097    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-28s8z\""
Sep 16 18:45:00.128200 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.128185    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:45:00.128307 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.128293    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:45:00.128364 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.128352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:45:00.128417 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.128393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:45:00.130476 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.130449    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:59 +0000 UTC" deadline="2028-02-19 00:19:23.653839393 +0000 UTC"
Sep 16 18:45:00.130579 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.130477    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12485h34m23.523366652s"
Sep 16 18:45:00.131689 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.131671    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal"]
Sep 16 18:45:00.131821 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.131807    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:45:00.142355 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.142329    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4gj8"
Sep 16 18:45:00.150542 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.150481    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v4gj8"
Sep 16 18:45:00.205075 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.205052    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:45:00.211477 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-ovnkube-script-lib\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.211563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-cnibin\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.211563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-kubelet\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.211563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl4ql\" (UniqueName: \"kubernetes.io/projected/15e98837-18f9-4305-a42e-695792e504f2-kube-api-access-cl4ql\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.211563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-socket-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.211684 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-system-cni-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.211684 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-socket-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.211684 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3780632e-fea7-4460-a811-203997052dcb-tmp-dir\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.211769 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d9b54d4-1563-4693-82de-c4496403392f-iptables-alerter-script\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.211805 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d9b54d4-1563-4693-82de-c4496403392f-host-slash\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.211805 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-multus-certs\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.211882 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjrvc\" (UniqueName: \"kubernetes.io/projected/2d9b54d4-1563-4693-82de-c4496403392f-kube-api-access-pjrvc\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.211882 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-slash\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.211882 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-log-socket\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.211986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-k8s-cni-cncf-io\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.211986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-device-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.211986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.211986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3780632e-fea7-4460-a811-203997052dcb-tmp-dir\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.211986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.211969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-kubernetes\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.212198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-device-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.212198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-ovn\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-env-overrides\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-netns\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.212198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212179    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/15e98837-18f9-4305-a42e-695792e504f2-serviceca\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-dbus\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-sys-fs\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cnibin\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-dbus\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-var-lib-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-sys-fs\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-kubelet-config\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cnibin\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdr8g\" (UniqueName: \"kubernetes.io/projected/3780632e-fea7-4460-a811-203997052dcb-kube-api-access-vdr8g\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-kubelet-config\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.212469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212440    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysctl-conf\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-systemd\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqdh\" (UniqueName: \"kubernetes.io/projected/3d97f496-2126-40cb-b3dd-763207a65c4b-kube-api-access-jtqdh\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-tuned\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212619    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-run-netns\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212625    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-cni-bin\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rrp\" (UniqueName: \"kubernetes.io/projected/77d082ca-6663-4b4a-9dbb-68e149d6fc10-kube-api-access-b6rrp\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f52644df-0002-449f-8b36-bc5a51714e1a-agent-certs\") pod \"konnectivity-agent-gxggm\" (UID: \"f52644df-0002-449f-8b36-bc5a51714e1a\") " pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-modprobe-d\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysconfig\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3780632e-fea7-4460-a811-203997052dcb-hosts-file\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-etc-selinux\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-host\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.212988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3780632e-fea7-4460-a811-203997052dcb-hosts-file\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-kubelet\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d97f496-2126-40cb-b3dd-763207a65c4b-ovn-node-metrics-cert\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91bad159-1568-4cf3-9695-1739ebaddfe0-cni-binary-copy\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-etc-selinux\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.212990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmb4f\" (UniqueName: \"kubernetes.io/projected/382ffe97-ecd3-4f98-92bd-47b589bcb233-kube-api-access-fmb4f\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-system-cni-dir\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.213051    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-daemon-config\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbw5\" (UniqueName: \"kubernetes.io/projected/91bad159-1568-4cf3-9695-1739ebaddfe0-kube-api-access-cwbw5\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.213107    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.713090481 +0000 UTC m=+2.017107447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f52644df-0002-449f-8b36-bc5a51714e1a-konnectivity-ca\") pod \"konnectivity-agent-gxggm\" (UID: \"f52644df-0002-449f-8b36-bc5a51714e1a\") " pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-systemd\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-var-lib-kubelet\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.213505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-cni-bin\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-socket-dir-parent\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-hostroot\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213209    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-conf-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mx2\" (UniqueName: \"kubernetes.io/projected/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-kube-api-access-f9mx2\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15e98837-18f9-4305-a42e-695792e504f2-host\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-cni-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-registration-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysctl-d\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-lib-modules\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e916f9-3065-4330-9e40-051ebc41a7db-tmp\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-cni-netd\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-cni-multus\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-etc-kubernetes\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-run\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlfv\" (UniqueName: \"kubernetes.io/projected/b4e916f9-3065-4330-9e40-051ebc41a7db-kube-api-access-gqlfv\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.214124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-ovnkube-config\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-sys\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-systemd-units\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-etc-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-node-log\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-system-cni-dir\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-registration-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-cni-binary-copy\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.214654 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.213776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/382ffe97-ecd3-4f98-92bd-47b589bcb233-kubelet-dir\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.220818 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.220791    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:45:00.224102 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.224074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmb4f\" (UniqueName: \"kubernetes.io/projected/382ffe97-ecd3-4f98-92bd-47b589bcb233-kube-api-access-fmb4f\") pod \"aws-ebs-csi-driver-node-phlw4\" (UID: \"382ffe97-ecd3-4f98-92bd-47b589bcb233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.224195 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.224113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdr8g\" (UniqueName: \"kubernetes.io/projected/3780632e-fea7-4460-a811-203997052dcb-kube-api-access-vdr8g\") pod \"node-resolver-ptxgf\" (UID: \"3780632e-fea7-4460-a811-203997052dcb\") " pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.224195 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.224113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mx2\" (UniqueName: \"kubernetes.io/projected/d1a7e37b-6e7b-4218-85be-fb1323ada9e0-kube-api-access-f9mx2\") pod \"multus-additional-cni-plugins-rj4pr\" (UID: \"d1a7e37b-6e7b-4218-85be-fb1323ada9e0\") " pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.266043 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.265996    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f1e051c1877a9e2bbc49221450c3da.slice/crio-3fe940afee01c2a41714e871b381ddb3429e0d9ae521caa736f07aeb6e314a61 WatchSource:0}: Error finding container 3fe940afee01c2a41714e871b381ddb3429e0d9ae521caa736f07aeb6e314a61: Status 404 returned error can't find the container with id 3fe940afee01c2a41714e871b381ddb3429e0d9ae521caa736f07aeb6e314a61
Sep 16 18:45:00.266245 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.266229    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee55f8a1340335ec8a8808d97d355d4.slice/crio-db7ae6565f74bd8dd4e2c5f1265ce55260929be036abbedbfb8811693bacd7ca WatchSource:0}: Error finding container db7ae6565f74bd8dd4e2c5f1265ce55260929be036abbedbfb8811693bacd7ca: Status 404 returned error can't find the container with id db7ae6565f74bd8dd4e2c5f1265ce55260929be036abbedbfb8811693bacd7ca
Sep 16 18:45:00.269976 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.269959    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:45:00.314178 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlfv\" (UniqueName: \"kubernetes.io/projected/b4e916f9-3065-4330-9e40-051ebc41a7db-kube-api-access-gqlfv\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.314178 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-ovnkube-config\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314207    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-sys\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-systemd-units\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-etc-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-node-log\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-sys\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-etc-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314346    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-node-log\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-systemd-units\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314913 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-ovnkube-script-lib\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.314913 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-cnibin\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.314913 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-cnibin\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315099 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.314481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-kubelet\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315283 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl4ql\" (UniqueName: \"kubernetes.io/projected/15e98837-18f9-4305-a42e-695792e504f2-kube-api-access-cl4ql\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.315364 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-system-cni-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315364 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d9b54d4-1563-4693-82de-c4496403392f-iptables-alerter-script\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.315364 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-kubelet\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315370    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d9b54d4-1563-4693-82de-c4496403392f-host-slash\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-multus-certs\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjrvc\" (UniqueName: \"kubernetes.io/projected/2d9b54d4-1563-4693-82de-c4496403392f-kube-api-access-pjrvc\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-slash\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-log-socket\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-system-cni-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315508 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-k8s-cni-cncf-io\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-kubernetes\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-ovnkube-script-lib\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-ovnkube-config\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-kubernetes\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-log-socket\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d9b54d4-1563-4693-82de-c4496403392f-host-slash\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-k8s-cni-cncf-io\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-multus-certs\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.315825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-slash\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-ovn\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315901    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-env-overrides\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-netns\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-ovn\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.315991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/15e98837-18f9-4305-a42e-695792e504f2-serviceca\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-var-lib-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysctl-conf\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-systemd\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d9b54d4-1563-4693-82de-c4496403392f-iptables-alerter-script\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.316276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqdh\" (UniqueName: \"kubernetes.io/projected/3d97f496-2126-40cb-b3dd-763207a65c4b-kube-api-access-jtqdh\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysctl-conf\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3d97f496-2126-40cb-b3dd-763207a65c4b-env-overrides\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-var-lib-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316466    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-run-netns\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-tuned\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-run-netns\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-cni-bin\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-cni-bin\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rrp\" (UniqueName: \"kubernetes.io/projected/77d082ca-6663-4b4a-9dbb-68e149d6fc10-kube-api-access-b6rrp\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-run-netns\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f52644df-0002-449f-8b36-bc5a51714e1a-agent-certs\") pod \"konnectivity-agent-gxggm\" (UID: \"f52644df-0002-449f-8b36-bc5a51714e1a\") " pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-modprobe-d\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysconfig\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-host\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-kubelet\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d97f496-2126-40cb-b3dd-763207a65c4b-ovn-node-metrics-cert\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.316924 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91bad159-1568-4cf3-9695-1739ebaddfe0-cni-binary-copy\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-daemon-config\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.316981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbw5\" (UniqueName: \"kubernetes.io/projected/91bad159-1568-4cf3-9695-1739ebaddfe0-kube-api-access-cwbw5\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f52644df-0002-449f-8b36-bc5a51714e1a-konnectivity-ca\") pod \"konnectivity-agent-gxggm\" (UID: \"f52644df-0002-449f-8b36-bc5a51714e1a\") " pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-systemd\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-var-lib-kubelet\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-cni-bin\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/15e98837-18f9-4305-a42e-695792e504f2-serviceca\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-socket-dir-parent\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317189    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-hostroot\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-socket-dir-parent\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-conf-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-systemd\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.317707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-conf-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.318337 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.317881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-systemd\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.318337 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91bad159-1568-4cf3-9695-1739ebaddfe0-cni-binary-copy\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.318337 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-daemon-config\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.318479 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318352    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-host\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.318479 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-cni-bin\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.318479 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318437    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-modprobe-d\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.318479 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-hostroot\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.318479 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-run-openvswitch\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.318713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318487    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysconfig\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.318713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.318713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15e98837-18f9-4305-a42e-695792e504f2-host\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.318713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-cni-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.318713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysctl-d\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.318713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-lib-modules\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318323    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-var-lib-kubelet\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318695    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e916f9-3065-4330-9e40-051ebc41a7db-tmp\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/15e98837-18f9-4305-a42e-695792e504f2-host\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-cni-netd\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-cni-multus\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.318897    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-host-var-lib-cni-multus\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.319058 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.318958    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.818939841 +0000 UTC m=+2.122956789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318897    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-etc-kubernetes\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-run\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.318541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-kubelet\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319233    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f52644df-0002-449f-8b36-bc5a51714e1a-konnectivity-ca\") pod \"konnectivity-agent-gxggm\" (UID: \"f52644df-0002-449f-8b36-bc5a51714e1a\") " pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-multus-cni-dir\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-sysctl-d\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-run\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3d97f496-2126-40cb-b3dd-763207a65c4b-host-cni-netd\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4e916f9-3065-4330-9e40-051ebc41a7db-lib-modules\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91bad159-1568-4cf3-9695-1739ebaddfe0-etc-kubernetes\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.320094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.319631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4e916f9-3065-4330-9e40-051ebc41a7db-etc-tuned\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.320714 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.320696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f52644df-0002-449f-8b36-bc5a51714e1a-agent-certs\") pod \"konnectivity-agent-gxggm\" (UID: \"f52644df-0002-449f-8b36-bc5a51714e1a\") " pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.321182 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.321169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e916f9-3065-4330-9e40-051ebc41a7db-tmp\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.321718 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.321703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3d97f496-2126-40cb-b3dd-763207a65c4b-ovn-node-metrics-cert\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.323632 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.323607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl4ql\" (UniqueName: \"kubernetes.io/projected/15e98837-18f9-4305-a42e-695792e504f2-kube-api-access-cl4ql\") pod \"node-ca-67d58\" (UID: \"15e98837-18f9-4305-a42e-695792e504f2\") " pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.323750 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.323734    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlfv\" (UniqueName: \"kubernetes.io/projected/b4e916f9-3065-4330-9e40-051ebc41a7db-kube-api-access-gqlfv\") pod \"tuned-djj5m\" (UID: \"b4e916f9-3065-4330-9e40-051ebc41a7db\") " pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.326646 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.326627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjrvc\" (UniqueName: \"kubernetes.io/projected/2d9b54d4-1563-4693-82de-c4496403392f-kube-api-access-pjrvc\") pod \"iptables-alerter-ttptx\" (UID: \"2d9b54d4-1563-4693-82de-c4496403392f\") " pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.326942 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.326926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqdh\" (UniqueName: \"kubernetes.io/projected/3d97f496-2126-40cb-b3dd-763207a65c4b-kube-api-access-jtqdh\") pod \"ovnkube-node-lwq4x\" (UID: \"3d97f496-2126-40cb-b3dd-763207a65c4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.329026 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.329002    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:00.329026 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.329021    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:00.329026 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.329030    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:00.329175 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.329109    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.829097746 +0000 UTC m=+2.133114703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:00.329781 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.329764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbw5\" (UniqueName: \"kubernetes.io/projected/91bad159-1568-4cf3-9695-1739ebaddfe0-kube-api-access-cwbw5\") pod \"multus-rnw6q\" (UID: \"91bad159-1568-4cf3-9695-1739ebaddfe0\") " pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.330725 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.330708    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rrp\" (UniqueName: \"kubernetes.io/projected/77d082ca-6663-4b4a-9dbb-68e149d6fc10-kube-api-access-b6rrp\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.431580 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.431525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4"
Sep 16 18:45:00.441291 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.441272    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382ffe97_ecd3_4f98_92bd_47b589bcb233.slice/crio-c73321f0be3d6c25fa1650a6c39eaba81ce8afdc58b12ebb68ce3960aac65d08 WatchSource:0}: Error finding container c73321f0be3d6c25fa1650a6c39eaba81ce8afdc58b12ebb68ce3960aac65d08: Status 404 returned error can't find the container with id c73321f0be3d6c25fa1650a6c39eaba81ce8afdc58b12ebb68ce3960aac65d08
Sep 16 18:45:00.449273 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.449257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ptxgf"
Sep 16 18:45:00.456223 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.456205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rj4pr"
Sep 16 18:45:00.456768 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.456745    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3780632e_fea7_4460_a811_203997052dcb.slice/crio-2c956b3fe610a24b6a949d7a958dd3b31794c1b422bbba19dd604063832d4a6e WatchSource:0}: Error finding container 2c956b3fe610a24b6a949d7a958dd3b31794c1b422bbba19dd604063832d4a6e: Status 404 returned error can't find the container with id 2c956b3fe610a24b6a949d7a958dd3b31794c1b422bbba19dd604063832d4a6e
Sep 16 18:45:00.463964 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.463946    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a7e37b_6e7b_4218_85be_fb1323ada9e0.slice/crio-fb3eec2fc7b4cf8e437ede6b2f64f9866772eba31f2706d7b336a0ed17c79a0c WatchSource:0}: Error finding container fb3eec2fc7b4cf8e437ede6b2f64f9866772eba31f2706d7b336a0ed17c79a0c: Status 404 returned error can't find the container with id fb3eec2fc7b4cf8e437ede6b2f64f9866772eba31f2706d7b336a0ed17c79a0c
Sep 16 18:45:00.486593 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.486575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rnw6q"
Sep 16 18:45:00.494183 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.494164    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bad159_1568_4cf3_9695_1739ebaddfe0.slice/crio-dfc3cc573f10027949494794901fa8733c341774a34826ecc3e654bead31d4e0 WatchSource:0}: Error finding container dfc3cc573f10027949494794901fa8733c341774a34826ecc3e654bead31d4e0: Status 404 returned error can't find the container with id dfc3cc573f10027949494794901fa8733c341774a34826ecc3e654bead31d4e0
Sep 16 18:45:00.500908 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.500892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:00.510035 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.510015    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52644df_0002_449f_8b36_bc5a51714e1a.slice/crio-3b621c34e8074ecd10774e31799ad9d01ad93ec59a4dcf5d5a031d073450b08c WatchSource:0}: Error finding container 3b621c34e8074ecd10774e31799ad9d01ad93ec59a4dcf5d5a031d073450b08c: Status 404 returned error can't find the container with id 3b621c34e8074ecd10774e31799ad9d01ad93ec59a4dcf5d5a031d073450b08c
Sep 16 18:45:00.514715 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.514702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-djj5m"
Sep 16 18:45:00.522013 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.521997    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e916f9_3065_4330_9e40_051ebc41a7db.slice/crio-4fa9ee3f631aa68a1e0401861d628954af6bec3f161ab7edee87fb8219b3abb3 WatchSource:0}: Error finding container 4fa9ee3f631aa68a1e0401861d628954af6bec3f161ab7edee87fb8219b3abb3: Status 404 returned error can't find the container with id 4fa9ee3f631aa68a1e0401861d628954af6bec3f161ab7edee87fb8219b3abb3
Sep 16 18:45:00.539922 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.539903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-67d58"
Sep 16 18:45:00.547191 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.547174    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e98837_18f9_4305_a42e_695792e504f2.slice/crio-b3eed6625e553570f34ac8fb37e38203c189bacfb8311a6316f817195b2a24fa WatchSource:0}: Error finding container b3eed6625e553570f34ac8fb37e38203c189bacfb8311a6316f817195b2a24fa: Status 404 returned error can't find the container with id b3eed6625e553570f34ac8fb37e38203c189bacfb8311a6316f817195b2a24fa
Sep 16 18:45:00.572439 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.572422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ttptx"
Sep 16 18:45:00.577223 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.577204    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:00.580268 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.580123    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9b54d4_1563_4693_82de_c4496403392f.slice/crio-50791d7244c9e8471a3a6bc91b0be571ac6305c766c14b0b7c48606af116ef1f WatchSource:0}: Error finding container 50791d7244c9e8471a3a6bc91b0be571ac6305c766c14b0b7c48606af116ef1f: Status 404 returned error can't find the container with id 50791d7244c9e8471a3a6bc91b0be571ac6305c766c14b0b7c48606af116ef1f
Sep 16 18:45:00.585097 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:00.585075    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d97f496_2126_40cb_b3dd_763207a65c4b.slice/crio-ae92392bc2764cf37b14780046345b1b6ab2b0c074180496708307b50dd45f33 WatchSource:0}: Error finding container ae92392bc2764cf37b14780046345b1b6ab2b0c074180496708307b50dd45f33: Status 404 returned error can't find the container with id ae92392bc2764cf37b14780046345b1b6ab2b0c074180496708307b50dd45f33
Sep 16 18:45:00.613031 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.613009    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:00.722100 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.722021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:00.722262 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.722196    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:00.722316 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.722265    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:01.722243661 +0000 UTC m=+3.026260620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:00.822602 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.822564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:00.822776 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.822700    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:00.822776 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.822759    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:01.822740751 +0000 UTC m=+3.126757699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:00.923518 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:00.923473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:00.923705 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.923640    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:00.923705 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.923661    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:00.923705 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.923672    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:00.923924 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:00.923728    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:01.923710331 +0000 UTC m=+3.227727279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:01.154106 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.153949    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:00 +0000 UTC" deadline="2028-03-25 17:35:42.148632703 +0000 UTC"
Sep 16 18:45:01.154106 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.154006    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13342h50m40.99463109s"
Sep 16 18:45:01.228004 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.227889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"ae92392bc2764cf37b14780046345b1b6ab2b0c074180496708307b50dd45f33"}
Sep 16 18:45:01.234290 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.234208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67d58" event={"ID":"15e98837-18f9-4305-a42e-695792e504f2","Type":"ContainerStarted","Data":"b3eed6625e553570f34ac8fb37e38203c189bacfb8311a6316f817195b2a24fa"}
Sep 16 18:45:01.240972 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.240885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" event={"ID":"bee55f8a1340335ec8a8808d97d355d4","Type":"ContainerStarted","Data":"db7ae6565f74bd8dd4e2c5f1265ce55260929be036abbedbfb8811693bacd7ca"}
Sep 16 18:45:01.256587 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.256552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ttptx" event={"ID":"2d9b54d4-1563-4693-82de-c4496403392f","Type":"ContainerStarted","Data":"50791d7244c9e8471a3a6bc91b0be571ac6305c766c14b0b7c48606af116ef1f"}
Sep 16 18:45:01.267044 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.267005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-djj5m" event={"ID":"b4e916f9-3065-4330-9e40-051ebc41a7db","Type":"ContainerStarted","Data":"4fa9ee3f631aa68a1e0401861d628954af6bec3f161ab7edee87fb8219b3abb3"}
Sep 16 18:45:01.275724 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.275602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gxggm" event={"ID":"f52644df-0002-449f-8b36-bc5a51714e1a","Type":"ContainerStarted","Data":"3b621c34e8074ecd10774e31799ad9d01ad93ec59a4dcf5d5a031d073450b08c"}
Sep 16 18:45:01.292535 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.292499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnw6q" event={"ID":"91bad159-1568-4cf3-9695-1739ebaddfe0","Type":"ContainerStarted","Data":"dfc3cc573f10027949494794901fa8733c341774a34826ecc3e654bead31d4e0"}
Sep 16 18:45:01.295341 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.295311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerStarted","Data":"fb3eec2fc7b4cf8e437ede6b2f64f9866772eba31f2706d7b336a0ed17c79a0c"}
Sep 16 18:45:01.301837 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.301807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptxgf" event={"ID":"3780632e-fea7-4460-a811-203997052dcb","Type":"ContainerStarted","Data":"2c956b3fe610a24b6a949d7a958dd3b31794c1b422bbba19dd604063832d4a6e"}
Sep 16 18:45:01.322062 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.320537    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4" event={"ID":"382ffe97-ecd3-4f98-92bd-47b589bcb233","Type":"ContainerStarted","Data":"c73321f0be3d6c25fa1650a6c39eaba81ce8afdc58b12ebb68ce3960aac65d08"}
Sep 16 18:45:01.324083 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.324061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"02f1e051c1877a9e2bbc49221450c3da","Type":"ContainerStarted","Data":"3fe940afee01c2a41714e871b381ddb3429e0d9ae521caa736f07aeb6e314a61"}
Sep 16 18:45:01.609020 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.608885    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:01.730287 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.729725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:01.730287 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.729900    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:01.730287 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.729965    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.729948186 +0000 UTC m=+5.033965146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:01.830756 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.830717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:01.830953 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.830893    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:01.831028 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.830960    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.830941749 +0000 UTC m=+5.134958708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:01.932713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:01.931981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:01.932713 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.932192    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:01.932713 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.932212    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:01.932713 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.932224    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:01.932713 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:01.932279    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:03.932262081 +0000 UTC m=+5.236279032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:02.154205 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:02.154127    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:00 +0000 UTC" deadline="2028-04-30 04:34:16.579385782 +0000 UTC"
Sep 16 18:45:02.154205 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:02.154166    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14193h49m14.42522407s"
Sep 16 18:45:02.217008 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:02.216147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:02.217008 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:02.216279    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:02.217008 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:02.216663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:02.217008 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:02.216764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:02.217008 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:02.216832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:02.217008 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:02.216916    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:02.897544 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:02.897310    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:45:03.746983 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:03.746904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:03.747417 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.747046    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.747417 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.747093    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.747079598 +0000 UTC m=+9.051096543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:03.848213 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:03.848173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:03.848436 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.848373    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.848436 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.848436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.848418142 +0000 UTC m=+9.152435099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:03.949562 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:03.949420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:03.949735 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.949632    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:03.949735 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.949654    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:03.949735 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.949666    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:03.949909 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:03.949754    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:07.949710914 +0000 UTC m=+9.253727863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:04.216023 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:04.215992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:04.216216 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:04.216102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:04.216216 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:04.216119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:04.216331 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:04.216218    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:04.216331 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:04.216309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:04.216424 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:04.216375    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:06.217028 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:06.216868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:06.217028 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:06.216910    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:06.217028 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:06.217021    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:06.217575 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:06.217100    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:06.217575 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:06.217154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:06.217575 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:06.217228    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:07.782050 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:07.782009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:07.782504 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.782162    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:07.782504 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.782217    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.782199837 +0000 UTC m=+17.086216786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:07.883443 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:07.883338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:07.883594 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.883506    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:07.883594 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.883568    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.883549771 +0000 UTC m=+17.187566718 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:07.984477 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:07.984402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:07.984632 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.984597    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:07.984632 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.984615    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:07.984632 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.984627    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:07.984782 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:07.984679    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.984663526 +0000 UTC m=+17.288680483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:08.216629 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:08.216595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:08.216798 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:08.216715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:08.216891 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:08.216721    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:08.216950 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:08.216891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:08.217002 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:08.216959    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:08.217002 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:08.216883    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:10.216181 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:10.215964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:10.216181 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:10.216010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:10.216181 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:10.216090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:10.216181 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:10.216098    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:10.216848 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:10.216211    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:10.216848 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:10.216265    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:12.216160 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:12.216128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:12.216160 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:12.216150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:12.216625 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:12.216234    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:12.216625 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:12.216276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:12.216625 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:12.216359    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:12.216625 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:12.216432    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:14.216485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:14.216449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:14.216485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:14.216466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:14.216979 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:14.216547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:14.216979 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:14.216607    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:14.216979 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:14.216707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:14.216979 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:14.216791    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:15.843016 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:15.842935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:15.843482 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:15.843101    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:15.843482 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:15.843172    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:31.843151474 +0000 UTC m=+33.147168424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:15.943413 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:15.943382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:15.943568 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:15.943537    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:15.943615 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:15.943609    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:31.943588259 +0000 UTC m=+33.247605204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:16.043839 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:16.043808    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:16.044032 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.043970    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:16.044032 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.043991    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:16.044032 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.044001    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:16.044162 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.044059    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:32.044043223 +0000 UTC m=+33.348060174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:16.216523 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:16.216450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:16.216523 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:16.216481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:16.216811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:16.216450    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:16.216811 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.216564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:16.216811 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.216645    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:16.216811 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:16.216706    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:18.216700 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:18.216659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:18.216700 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:18.216688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:18.217254 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:18.216696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:18.217254 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:18.216779    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:18.217254 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:18.216851    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:18.217254 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:18.216963    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:20.216944 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.216588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:20.217475 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.216630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:20.217475 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:20.217054    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:20.217475 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.216631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:20.217475 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:20.217141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:20.217475 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:20.217177    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:20.365597 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.365526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-djj5m" event={"ID":"b4e916f9-3065-4330-9e40-051ebc41a7db","Type":"ContainerStarted","Data":"93abaea7361a89663c825969ccab34f98c85a71ab1dece6f9014a18bd4424d6e"}
Sep 16 18:45:20.367107 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.367082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gxggm" event={"ID":"f52644df-0002-449f-8b36-bc5a51714e1a","Type":"ContainerStarted","Data":"e15f369812d743883d3e561959513f1d5bf857f7555c8e3febb72aa2e6622569"}
Sep 16 18:45:20.368724 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.368655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rnw6q" event={"ID":"91bad159-1568-4cf3-9695-1739ebaddfe0","Type":"ContainerStarted","Data":"be04737e6ddd0af437e74606f7b18dcda919d368a1e11f0b220595333a6c7272"}
Sep 16 18:45:20.370470 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.370447    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1a7e37b-6e7b-4218-85be-fb1323ada9e0" containerID="8b23ec3b0c6dcdeebddacea75d0b325a6d184533c26988fc97bf4ed8517cf421" exitCode=0
Sep 16 18:45:20.370556 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.370519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerDied","Data":"8b23ec3b0c6dcdeebddacea75d0b325a6d184533c26988fc97bf4ed8517cf421"}
Sep 16 18:45:20.372189 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.372165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ptxgf" event={"ID":"3780632e-fea7-4460-a811-203997052dcb","Type":"ContainerStarted","Data":"73b0ca4d33b1ca64d324bac62a7fcee4038dce50b1230fd9af0a282648f03a77"}
Sep 16 18:45:20.373769 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.373742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4" event={"ID":"382ffe97-ecd3-4f98-92bd-47b589bcb233","Type":"ContainerStarted","Data":"5cd1b7b5a5f8d5ff1c53009380b1d248265dc5a74c518216881dd521259ccdc7"}
Sep 16 18:45:20.375334 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.375306    2581 generic.go:356] "Generic (PLEG): container finished" podID="02f1e051c1877a9e2bbc49221450c3da" containerID="b97a758fa3ab5c0b5cef962d65da36242778fd3a1c08bcbf4ea3114ac5d1ef07" exitCode=0
Sep 16 18:45:20.375421 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.375380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"02f1e051c1877a9e2bbc49221450c3da","Type":"ContainerDied","Data":"b97a758fa3ab5c0b5cef962d65da36242778fd3a1c08bcbf4ea3114ac5d1ef07"}
Sep 16 18:45:20.378946 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.378926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:45:20.379307 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379287    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d97f496-2126-40cb-b3dd-763207a65c4b" containerID="009f3c726d41ed2af42ea49e6728f4e3b5c848cb098ea26950893fe8458121ae" exitCode=1
Sep 16 18:45:20.379373 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"8e3cbb2e9d856b650157728df9c48908421734ccd1f738cfdb0904d9c82660b7"}
Sep 16 18:45:20.379437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"95073d1a704a254fd74303aa52f4d8eaba8a6a47dfdbd809d2efc08deb7b0537"}
Sep 16 18:45:20.379535 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"c545573810349497d08e1d85886a23b53a3be6b1af17fe30e1d581259c350f43"}
Sep 16 18:45:20.379593 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"ea15f246259bcb6a4cf42715a1a73cbc7dbb2aa30616aa0c6e8266a64bdd87fb"}
Sep 16 18:45:20.379629 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerDied","Data":"009f3c726d41ed2af42ea49e6728f4e3b5c848cb098ea26950893fe8458121ae"}
Sep 16 18:45:20.379667 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.379633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"aa5a9a5287be18af8bc2596a49be2e8a3c64ab0035b94416b9dd0fb23b9bf948"}
Sep 16 18:45:20.380785 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.380766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-67d58" event={"ID":"15e98837-18f9-4305-a42e-695792e504f2","Type":"ContainerStarted","Data":"26553be8f64996af1f6c82ef110594ac859fe7809d2f9e000e553fa926734e62"}
Sep 16 18:45:20.382257 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.382204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" event={"ID":"bee55f8a1340335ec8a8808d97d355d4","Type":"ContainerStarted","Data":"8bbf22163cdb5855bfd3aab5731ab12ee71d09231d9899ff9ca02cdaff32e08c"}
Sep 16 18:45:20.387612 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.387567    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-djj5m" podStartSLOduration=2.349932006 podStartE2EDuration="21.387552333s" totalImagesPullingTime="19.037620327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.387111234 +0000 UTC m=+21.691128201" watchObservedRunningTime="2026-09-16 18:45:20.387552333 +0000 UTC m=+21.691569301"
Sep 16 18:45:20.403062 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.403020    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-67d58" podStartSLOduration=2.392169113 podStartE2EDuration="21.403004931s" totalImagesPullingTime="19.010835818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.402146485 +0000 UTC m=+21.706163453" watchObservedRunningTime="2026-09-16 18:45:20.403004931 +0000 UTC m=+21.707021899"
Sep 16 18:45:20.449797 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.449763    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rnw6q" podStartSLOduration=2.268386186 podStartE2EDuration="21.449750218s" totalImagesPullingTime="19.181364032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.449242838 +0000 UTC m=+21.753259805" watchObservedRunningTime="2026-09-16 18:45:20.449750218 +0000 UTC m=+21.753767184"
Sep 16 18:45:20.465707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.465669    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-194.ec2.internal" podStartSLOduration=20.465657243 podStartE2EDuration="20.465657243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.465417039 +0000 UTC m=+21.769434006" watchObservedRunningTime="2026-09-16 18:45:20.465657243 +0000 UTC m=+21.769674210"
Sep 16 18:45:20.480243 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.480212    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ptxgf" podStartSLOduration=2.408090689 podStartE2EDuration="21.480199517s" totalImagesPullingTime="19.072108828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.479808933 +0000 UTC m=+21.783825920" watchObservedRunningTime="2026-09-16 18:45:20.480199517 +0000 UTC m=+21.784216484"
Sep 16 18:45:20.518194 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:20.518150    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gxggm" podStartSLOduration=2.46835343 podStartE2EDuration="21.51813311s" totalImagesPullingTime="19.04977968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:20.517773631 +0000 UTC m=+21.821790597" watchObservedRunningTime="2026-09-16 18:45:20.51813311 +0000 UTC m=+21.822150079"
Sep 16 18:45:21.078385 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.078358    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:21.168075 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.167979    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:21.078381371Z","UUID":"73a5cdd2-695a-470d-909e-60826ec363dc","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:21.169951 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.169933    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:21.170053 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.169964    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:21.387374 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.387338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ttptx" event={"ID":"2d9b54d4-1563-4693-82de-c4496403392f","Type":"ContainerStarted","Data":"866e2536a3f719d5563815fa9a0529e06e01b9efacad97eb18d993c58bfc97e3"}
Sep 16 18:45:21.389464 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.389436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4" event={"ID":"382ffe97-ecd3-4f98-92bd-47b589bcb233","Type":"ContainerStarted","Data":"6761c1efc52e0a5ba8c38a5e826b416ec0637fa17b29567091807ff24d1b67bc"}
Sep 16 18:45:21.391492 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.391339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" event={"ID":"02f1e051c1877a9e2bbc49221450c3da","Type":"ContainerStarted","Data":"20a0dd380ff5244294482bdb8a6033e54faeaf87c8b479f9636c9a9bfb507fb3"}
Sep 16 18:45:21.402522 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.402429    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ttptx" podStartSLOduration=3.419946604 podStartE2EDuration="22.402411953s" totalImagesPullingTime="18.982465349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:21.402042389 +0000 UTC m=+22.706059358" watchObservedRunningTime="2026-09-16 18:45:21.402411953 +0000 UTC m=+22.706428921"
Sep 16 18:45:21.417137 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.417098    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-194.ec2.internal" podStartSLOduration=21.417084506 podStartE2EDuration="21.417084506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:21.416493112 +0000 UTC m=+22.720510079" watchObservedRunningTime="2026-09-16 18:45:21.417084506 +0000 UTC m=+22.721101475"
Sep 16 18:45:21.581896 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.581844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:21.582663 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:21.582644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:22.216557 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.216530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:22.216652 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:22.216636    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:22.220935 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.217777    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:22.220935 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:22.217983    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:22.220935 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.218029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:22.220935 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:22.218133    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:22.398074 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.397637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4" event={"ID":"382ffe97-ecd3-4f98-92bd-47b589bcb233","Type":"ContainerStarted","Data":"a7e4816ab3812e609660b2f7f076abecccaa2e411ada384c29dd440186839ba4"}
Sep 16 18:45:22.401626 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.401604    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:45:22.403619 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.402351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"708c193791975a8489ab24bf0b91c796dcd6ffb9eb95e38e86969ed75de562f8"}
Sep 16 18:45:22.414879 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:22.414815    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-phlw4" podStartSLOduration=2.426891674 podStartE2EDuration="23.414795773s" totalImagesPullingTime="20.987904099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:22.414102844 +0000 UTC m=+23.718119814" watchObservedRunningTime="2026-09-16 18:45:22.414795773 +0000 UTC m=+23.718812741"
Sep 16 18:45:23.404213 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:23.404004    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:24.216926 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:24.216893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:24.217164 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:24.216893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:24.217164 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:24.217005    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:24.217164 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:24.216904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:24.217164 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:24.217078    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:24.217336 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:24.217212    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:25.410449 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.410255    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1a7e37b-6e7b-4218-85be-fb1323ada9e0" containerID="72643c097e6377b6036fcc3d0b230b811f044dc1a17e0fa6c0531ee4e779d9be" exitCode=0
Sep 16 18:45:25.410902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.410338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerDied","Data":"72643c097e6377b6036fcc3d0b230b811f044dc1a17e0fa6c0531ee4e779d9be"}
Sep 16 18:45:25.414329 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.414311    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:45:25.414710 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.414683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"432a142fb31fa65152f2932986fc192ee5e7d027338464ddf3343c86eec24117"}
Sep 16 18:45:25.417869 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.415817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:25.417869 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.415960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:25.417869 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.416743    2581 scope.go:117] "RemoveContainer" containerID="009f3c726d41ed2af42ea49e6728f4e3b5c848cb098ea26950893fe8458121ae"
Sep 16 18:45:25.434483 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:25.434463    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:26.215953 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.215928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:26.216084 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.215929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:26.216084 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:26.216033    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:26.216182 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.215929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:26.216265 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:26.216153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:26.216309 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:26.216266    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:26.418530 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.418452    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1a7e37b-6e7b-4218-85be-fb1323ada9e0" containerID="7441d986951ceeb68ba21518959b3e4699f0525e175d0ecbe32019f2042f3a4b" exitCode=0
Sep 16 18:45:26.419236 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.418541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerDied","Data":"7441d986951ceeb68ba21518959b3e4699f0525e175d0ecbe32019f2042f3a4b"}
Sep 16 18:45:26.422394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.422377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:45:26.422712 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.422695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" event={"ID":"3d97f496-2126-40cb-b3dd-763207a65c4b","Type":"ContainerStarted","Data":"bdc9f58e391a697ad0a27f6a77f7b0ec764b535da15a443ab6f6bc28620b90de"}
Sep 16 18:45:26.422892 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.422875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:26.437422 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.437405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:26.471405 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.471360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x" podStartSLOduration=8.497198862 podStartE2EDuration="27.471348055s" totalImagesPullingTime="18.974149193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:26.471292603 +0000 UTC m=+27.775309571" watchObservedRunningTime="2026-09-16 18:45:26.471348055 +0000 UTC m=+27.775365022"
Sep 16 18:45:26.481724 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.481703    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qb9wt"]
Sep 16 18:45:26.481873 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.481841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:26.481985 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:26.481965    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:26.484911 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.484882    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hxpgk"]
Sep 16 18:45:26.485004 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.484978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:26.485120 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:26.485076    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:26.485552 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.485533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgznf"]
Sep 16 18:45:26.485644 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:26.485630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:26.485742 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:26.485725    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:27.427437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:27.427351    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1a7e37b-6e7b-4218-85be-fb1323ada9e0" containerID="2592fb88ad384aa502f5612cf15ef370c460f1d9c493c51080d6d6403ccccb53" exitCode=0
Sep 16 18:45:27.427437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:27.427418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerDied","Data":"2592fb88ad384aa502f5612cf15ef370c460f1d9c493c51080d6d6403ccccb53"}
Sep 16 18:45:28.216084 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:28.216048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:28.216084 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:28.216079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:28.216323 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:28.216079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:28.216323 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:28.216182    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:28.216323 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:28.216263    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:28.216435 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:28.216355    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:30.155743 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:30.155702    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:30.156409 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:30.155842    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:45:30.156713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:30.156577    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gxggm"
Sep 16 18:45:30.216216 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:30.216155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:30.216369 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:30.216272    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:30.216369 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:30.216276    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:30.216369 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:30.216307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:30.216529 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:30.216390    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:30.216529 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:30.216478    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:31.852936 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:31.852909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:31.853260 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:31.853043    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:31.853260 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:31.853112    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret podName:6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:03.853092364 +0000 UTC m=+65.157109310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret") pod "global-pull-secret-syncer-hxpgk" (UID: "6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:31.953485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:31.953456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:31.953618 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:31.953592    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:31.953666 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:31.953645    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:03.953630569 +0000 UTC m=+65.257647528 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:32.053804 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.053756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:32.053919 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.053910    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:32.053969 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.053925    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:32.053969 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.053934    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hqh2d for pod openshift-network-diagnostics/network-check-target-qb9wt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:32.054044 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.053976    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d podName:ca425601-c108-47c8-b580-06103b15bb19 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.053963608 +0000 UTC m=+65.357980552 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hqh2d" (UniqueName: "kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d") pod "network-check-target-qb9wt" (UID: "ca425601-c108-47c8-b580-06103b15bb19") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:32.215970 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.215939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:32.215970 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.215953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:32.216158 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.216050    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tgznf" podUID="77d082ca-6663-4b4a-9dbb-68e149d6fc10"
Sep 16 18:45:32.216158 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.216117    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hxpgk" podUID="6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9"
Sep 16 18:45:32.216226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.216161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:32.216261 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.216229    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qb9wt" podUID="ca425601-c108-47c8-b580-06103b15bb19"
Sep 16 18:45:32.441146 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.441115    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1a7e37b-6e7b-4218-85be-fb1323ada9e0" containerID="407eec05e9eca87a38c3290bf5b4e49e981be2dc1539cb981c5c1f3a9d4d6be0" exitCode=0
Sep 16 18:45:32.441285 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.441167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerDied","Data":"407eec05e9eca87a38c3290bf5b4e49e981be2dc1539cb981c5c1f3a9d4d6be0"}
Sep 16 18:45:32.458410 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.458387    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-194.ec2.internal" event="NodeReady"
Sep 16 18:45:32.458503 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.458480    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:32.491679 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.491655    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-89845797-2jj42"]
Sep 16 18:45:32.510946 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.510920    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pkbvg"]
Sep 16 18:45:32.511097 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.511065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.513719 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.513696    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bk56d\""
Sep 16 18:45:32.514327 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.514177    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:45:32.514327 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.514197    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:45:32.514327 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.514290    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:45:32.521382 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.521357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:45:32.527396 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.527378    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-89845797-2jj42"]
Sep 16 18:45:32.527480 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.527406    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hwz4m"]
Sep 16 18:45:32.527519 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.527485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.530058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.530038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:32.530151 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.530048    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:32.530151 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.530081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vf4dn\""
Sep 16 18:45:32.545350 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.545334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkbvg"]
Sep 16 18:45:32.545438 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.545362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwz4m"]
Sep 16 18:45:32.545438 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.545420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.547622 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.547601    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:32.547750 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.547657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s4v9d\""
Sep 16 18:45:32.547825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.547814    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:32.547877 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.547839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:32.658468 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658598 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-certificates\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658598 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5232b259-8d8e-4a9e-bbec-26720ec51c17-config-volume\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.658598 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.658598 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwl5k\" (UniqueName: \"kubernetes.io/projected/5232b259-8d8e-4a9e-bbec-26720ec51c17-kube-api-access-gwl5k\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.658598 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fab74c7b-6b13-4e5e-a494-93793cda1407-cert\") pod \"ingress-canary-hwz4m\" (UID: \"fab74c7b-6b13-4e5e-a494-93793cda1407\") " pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-ca-trust-extracted\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5232b259-8d8e-4a9e-bbec-26720ec51c17-tmp-dir\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-trusted-ca\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-installation-pull-secrets\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-bound-sa-token\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scgsm\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-kube-api-access-scgsm\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-image-registry-private-configuration\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.658885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.658883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7g42\" (UniqueName: \"kubernetes.io/projected/fab74c7b-6b13-4e5e-a494-93793cda1407-kube-api-access-d7g42\") pod \"ingress-canary-hwz4m\" (UID: \"fab74c7b-6b13-4e5e-a494-93793cda1407\") " pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.759715 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-bound-sa-token\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.759715 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scgsm\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-kube-api-access-scgsm\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-image-registry-private-configuration\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7g42\" (UniqueName: \"kubernetes.io/projected/fab74c7b-6b13-4e5e-a494-93793cda1407-kube-api-access-d7g42\") pod \"ingress-canary-hwz4m\" (UID: \"fab74c7b-6b13-4e5e-a494-93793cda1407\") " pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759835    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-certificates\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5232b259-8d8e-4a9e-bbec-26720ec51c17-config-volume\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwl5k\" (UniqueName: \"kubernetes.io/projected/5232b259-8d8e-4a9e-bbec-26720ec51c17-kube-api-access-gwl5k\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fab74c7b-6b13-4e5e-a494-93793cda1407-cert\") pod \"ingress-canary-hwz4m\" (UID: \"fab74c7b-6b13-4e5e-a494-93793cda1407\") " pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.759944    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.759966    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-ca-trust-extracted\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.759994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5232b259-8d8e-4a9e-bbec-26720ec51c17-tmp-dir\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.760045 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.760031    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.260004257 +0000 UTC m=+34.564021215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.760072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-trusted-ca\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.760105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-installation-pull-secrets\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.760267    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:32.760327    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:33.260302571 +0000 UTC m=+34.564319530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.760358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5232b259-8d8e-4a9e-bbec-26720ec51c17-tmp-dir\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.760474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5232b259-8d8e-4a9e-bbec-26720ec51c17-config-volume\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.760621 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.760542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-certificates\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.761032 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.760694    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-ca-trust-extracted\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.761125 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.761102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-trusted-ca\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.763951 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.763929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-image-registry-private-configuration\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.764065 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.763951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fab74c7b-6b13-4e5e-a494-93793cda1407-cert\") pod \"ingress-canary-hwz4m\" (UID: \"fab74c7b-6b13-4e5e-a494-93793cda1407\") " pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.764065 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.763951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-installation-pull-secrets\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.772562 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.772542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwl5k\" (UniqueName: \"kubernetes.io/projected/5232b259-8d8e-4a9e-bbec-26720ec51c17-kube-api-access-gwl5k\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:32.772686 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.772642    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-bound-sa-token\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.772686 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.772681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scgsm\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-kube-api-access-scgsm\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:32.772957 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.772939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7g42\" (UniqueName: \"kubernetes.io/projected/fab74c7b-6b13-4e5e-a494-93793cda1407-kube-api-access-d7g42\") pod \"ingress-canary-hwz4m\" (UID: \"fab74c7b-6b13-4e5e-a494-93793cda1407\") " pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:32.860955 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:32.860927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hwz4m"
Sep 16 18:45:33.149957 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:33.149886    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hwz4m"]
Sep 16 18:45:33.153496 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:45:33.153469    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab74c7b_6b13_4e5e_a494_93793cda1407.slice/crio-367cd79eb87da66df3822542774b67f9ddba65dd0cac42062326740c51e9d9c8 WatchSource:0}: Error finding container 367cd79eb87da66df3822542774b67f9ddba65dd0cac42062326740c51e9d9c8: Status 404 returned error can't find the container with id 367cd79eb87da66df3822542774b67f9ddba65dd0cac42062326740c51e9d9c8
Sep 16 18:45:33.265083 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:33.264928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:33.265221 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:33.265100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:33.265221 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:33.265063    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:33.265221 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:33.265157    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:45:33.265221 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:33.265185    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:33.265221 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:33.265212    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.265196972 +0000 UTC m=+35.569213932 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:45:33.265221 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:33.265225    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:34.265219104 +0000 UTC m=+35.569236048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:33.444451 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:33.444425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwz4m" event={"ID":"fab74c7b-6b13-4e5e-a494-93793cda1407","Type":"ContainerStarted","Data":"367cd79eb87da66df3822542774b67f9ddba65dd0cac42062326740c51e9d9c8"}
Sep 16 18:45:33.447487 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:33.447464    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1a7e37b-6e7b-4218-85be-fb1323ada9e0" containerID="ee3580ae0b97a94c590ee3494b68cfc7cab62d0e31dfebdfc22384beb1d52dd3" exitCode=0
Sep 16 18:45:33.447593 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:33.447512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerDied","Data":"ee3580ae0b97a94c590ee3494b68cfc7cab62d0e31dfebdfc22384beb1d52dd3"}
Sep 16 18:45:34.217020 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.216986    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:45:34.217693 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.217029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:45:34.217693 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.217034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:45:34.219827 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.219800    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:34.221131 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.220976    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:34.221131 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.220980    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m77pk\""
Sep 16 18:45:34.221131 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.221007    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:34.221131 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.221072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpzh6\""
Sep 16 18:45:34.221385 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.221188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:34.273375 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.273349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:34.273485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.273404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:34.273533 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:34.273498    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:34.273533 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:34.273515    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:45:34.273616 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:34.273553    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:34.273667 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:34.273602    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.273557479 +0000 UTC m=+37.577574440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:45:34.273667 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:34.273639    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:36.27362872 +0000 UTC m=+37.577645665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:34.454193 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.454163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" event={"ID":"d1a7e37b-6e7b-4218-85be-fb1323ada9e0","Type":"ContainerStarted","Data":"95feaa59165c169d9465f32bfd46bb6f284df25cb0616d7c46c9a6af0666974d"}
Sep 16 18:45:34.469153 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.469078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:34.475217 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:34.475176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rj4pr" podStartSLOduration=6.614982277 podStartE2EDuration="35.475162739s" totalImagesPullingTime="28.860180462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.474730097 +0000 UTC m=+35.778747064" watchObservedRunningTime="2026-09-16 18:45:34.475162739 +0000 UTC m=+35.779179706"
Sep 16 18:45:36.023611 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:36.023583    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:36.290010 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:36.289923    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:36.290010 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:36.289970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:36.290227 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:36.290074    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:36.290227 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:36.290100    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:36.290227 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:36.290120    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:45:36.290227 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:36.290130    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.290114337 +0000 UTC m=+41.594131299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:36.290227 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:36.290171    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.290153326 +0000 UTC m=+41.594170287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:45:36.460258 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:36.460227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hwz4m" event={"ID":"fab74c7b-6b13-4e5e-a494-93793cda1407","Type":"ContainerStarted","Data":"b163060802bb5cadb75f4ce1e2411fb6d96e0f8217157b58fcb1354deb694f68"}
Sep 16 18:45:36.476786 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:36.476721    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hwz4m" podStartSLOduration=2.180067746 podStartE2EDuration="4.476701941s" totalImagesPullingTime="2.296634195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:36.476468333 +0000 UTC m=+37.780485301" watchObservedRunningTime="2026-09-16 18:45:36.476701941 +0000 UTC m=+37.780718908"
Sep 16 18:45:38.049903 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:38.049844    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:38.111345 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:38.111313    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:40.320462 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:40.320418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:40.320462 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:40.320470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:40.320897 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:40.320576    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:40.320897 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:40.320579    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:40.320897 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:40.320600    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:45:40.320897 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:40.320630    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.320614977 +0000 UTC m=+49.624631942 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:40.320897 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:40.320643    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.320637509 +0000 UTC m=+49.624654453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:45:43.660283 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:43.660245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:43.788271 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:43.788237    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:43.847271 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:43.847238    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:43.928904 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:43.928802    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:44.626075 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:44.626043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:44.726322 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:44.726286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.025202 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:45.025167    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.090080 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:45.090044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:45.836929 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:45.836892    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:46.233569 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:46.233539    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:46.330792 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:46.330765    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:48.374547 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:48.374509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:45:48.374994 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:48.374553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:45:48.374994 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:48.374661    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:48.374994 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:48.374687    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:48.374994 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:48.374713    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:45:48.374994 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:48.374727    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.374710914 +0000 UTC m=+65.678727873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:48.374994 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:45:48.374778    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.37475963 +0000 UTC m=+65.678776588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:45:53.581303 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:53.581263    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:58.440401 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:58.440373    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lwq4x"
Sep 16 18:45:58.595165 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:58.595121    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:58.695249 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:58.695159    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:58.947343 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:58.947253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:59.063156 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:45:59.063127    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:01.404047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:01.404007    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:02.909530 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:02.909499    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:03.017740 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.017709    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:03.887521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.887489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:46:03.890462 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.890445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:46:03.900822 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.900797    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9-original-pull-secret\") pod \"global-pull-secret-syncer-hxpgk\" (UID: \"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9\") " pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:46:03.929837 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.929817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hxpgk"
Sep 16 18:46:03.988640 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.988608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:46:03.991698 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:03.991673    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:46:03.998987 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:03.998965    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:46:03.999079 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:03.999038    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs podName:77d082ca-6663-4b4a-9dbb-68e149d6fc10 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:07.999020755 +0000 UTC m=+129.303037700 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs") pod "network-metrics-daemon-tgznf" (UID: "77d082ca-6663-4b4a-9dbb-68e149d6fc10") : secret "metrics-daemon-secret" not found
Sep 16 18:46:04.089130 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.089102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:46:04.092320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.092304    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:46:04.101988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.101972    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:46:04.112318 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.112296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqh2d\" (UniqueName: \"kubernetes.io/projected/ca425601-c108-47c8-b580-06103b15bb19-kube-api-access-hqh2d\") pod \"network-check-target-qb9wt\" (UID: \"ca425601-c108-47c8-b580-06103b15bb19\") " pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:46:04.193370 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.193346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hxpgk"]
Sep 16 18:46:04.196143 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:04.196122    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f52e1e8_f991_4bb7_8e84_1043fb9d5ab9.slice/crio-8236290f33530ba4a9891209f3c821fc5100b1dd7064805b247d66d98fcefa43 WatchSource:0}: Error finding container 8236290f33530ba4a9891209f3c821fc5100b1dd7064805b247d66d98fcefa43: Status 404 returned error can't find the container with id 8236290f33530ba4a9891209f3c821fc5100b1dd7064805b247d66d98fcefa43
Sep 16 18:46:04.245568 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.245546    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jpzh6\""
Sep 16 18:46:04.253832 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.253805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:46:04.395731 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.395691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:46:04.395952 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.395748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:46:04.395952 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:04.395911    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:46:04.395952 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:04.395916    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:46:04.395952 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:04.395941    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-89845797-2jj42: secret "image-registry-tls" not found
Sep 16 18:46:04.396132 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:04.395965    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls podName:5232b259-8d8e-4a9e-bbec-26720ec51c17 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:36.395952286 +0000 UTC m=+97.699969231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls") pod "dns-default-pkbvg" (UID: "5232b259-8d8e-4a9e-bbec-26720ec51c17") : secret "dns-default-metrics-tls" not found
Sep 16 18:46:04.396132 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:04.396000    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls podName:e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:36.395981341 +0000 UTC m=+97.699998287 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls") pod "image-registry-89845797-2jj42" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2") : secret "image-registry-tls" not found
Sep 16 18:46:04.521423 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.521340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qb9wt"]
Sep 16 18:46:04.523952 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:04.523925    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca425601_c108_47c8_b580_06103b15bb19.slice/crio-71082730014b5342c5f00cd9a28f229f0de56242afff318f52caeeae2a286a0d WatchSource:0}: Error finding container 71082730014b5342c5f00cd9a28f229f0de56242afff318f52caeeae2a286a0d: Status 404 returned error can't find the container with id 71082730014b5342c5f00cd9a28f229f0de56242afff318f52caeeae2a286a0d
Sep 16 18:46:04.527546 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.527507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qb9wt" event={"ID":"ca425601-c108-47c8-b580-06103b15bb19","Type":"ContainerStarted","Data":"71082730014b5342c5f00cd9a28f229f0de56242afff318f52caeeae2a286a0d"}
Sep 16 18:46:04.528638 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:04.528593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hxpgk" event={"ID":"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9","Type":"ContainerStarted","Data":"8236290f33530ba4a9891209f3c821fc5100b1dd7064805b247d66d98fcefa43"}
Sep 16 18:46:09.546805 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:09.546757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qb9wt" event={"ID":"ca425601-c108-47c8-b580-06103b15bb19","Type":"ContainerStarted","Data":"85edd7c3811c9c63ffc193296b8aff7a3c234109892fb07bb7b2ef2b4b2ee3d8"}
Sep 16 18:46:09.547313 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:09.546844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:46:09.548206 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:09.548185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hxpgk" event={"ID":"6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9","Type":"ContainerStarted","Data":"bddb08b5c3b01c6839be789ab0220a0a1f4a19e636cd729e7ca498402f56f04c"}
Sep 16 18:46:09.562405 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:09.562336    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qb9wt" podStartSLOduration=66.360780632 podStartE2EDuration="1m10.562322284s" totalImagesPullingTime="4.201541652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.561922228 +0000 UTC m=+70.865939197" watchObservedRunningTime="2026-09-16 18:46:09.562322284 +0000 UTC m=+70.866339252"
Sep 16 18:46:09.576357 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:09.576305    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hxpgk" podStartSLOduration=66.04379328 podStartE2EDuration="1m10.576293032s" totalImagesPullingTime="4.532499752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.575947879 +0000 UTC m=+70.879964846" watchObservedRunningTime="2026-09-16 18:46:09.576293032 +0000 UTC m=+70.880309998"
Sep 16 18:46:16.529702 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.529669    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85465d975b-5frng"]
Sep 16 18:46:16.533019 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.533004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.535757 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.535735    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ht55s\""
Sep 16 18:46:16.536110 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.536094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 18:46:16.536393 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.536378    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:46:16.536450 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.536399    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 18:46:16.539305 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.539279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 18:46:16.540820 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.539524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 18:46:16.540820 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.539883    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:46:16.547351 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.547326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85465d975b-5frng"]
Sep 16 18:46:16.580475 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.580442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxktg\" (UniqueName: \"kubernetes.io/projected/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-kube-api-access-hxktg\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.580680 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.580485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-stats-auth\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.580680 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.580603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-default-certificate\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.580680 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.580636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.580680 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.580660    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.681908 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.681870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-default-certificate\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.681908 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.681909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.682166 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.681925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.682166 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.681949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxktg\" (UniqueName: \"kubernetes.io/projected/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-kube-api-access-hxktg\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.682166 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.681972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-stats-auth\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.682166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:16.682062    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.18203899 +0000 UTC m=+78.486055953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:16.682166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:16.682071    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:16.682166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:16.682133    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:17.182118535 +0000 UTC m=+78.486135497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:16.684326 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.684302    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-default-certificate\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.684419 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.684348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-stats-auth\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.690550 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.690528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxktg\" (UniqueName: \"kubernetes.io/projected/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-kube-api-access-hxktg\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:16.727785 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.727755    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-krl58"]
Sep 16 18:46:16.730930 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.730915    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.737573 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.737549    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:46:16.737678 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.737588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cmhkl\""
Sep 16 18:46:16.737951 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.737936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:46:16.738023 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.737941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:46:16.738080 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.738035    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:46:16.742162 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.742139    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-krl58"]
Sep 16 18:46:16.743509 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.743491    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:46:16.782343 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.782258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c6d5ee-8126-4260-88a3-7920cf4cb532-config\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.782470 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.782374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prrsw\" (UniqueName: \"kubernetes.io/projected/b1c6d5ee-8126-4260-88a3-7920cf4cb532-kube-api-access-prrsw\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.782470 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.782397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6d5ee-8126-4260-88a3-7920cf4cb532-trusted-ca\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.782470 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.782417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1c6d5ee-8126-4260-88a3-7920cf4cb532-serving-cert\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.882840 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.882813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c6d5ee-8126-4260-88a3-7920cf4cb532-config\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.882987 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.882910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prrsw\" (UniqueName: \"kubernetes.io/projected/b1c6d5ee-8126-4260-88a3-7920cf4cb532-kube-api-access-prrsw\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.882987 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.882929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6d5ee-8126-4260-88a3-7920cf4cb532-trusted-ca\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.882987 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.882945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1c6d5ee-8126-4260-88a3-7920cf4cb532-serving-cert\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.883519 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.883496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b1c6d5ee-8126-4260-88a3-7920cf4cb532-config\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.883704 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.883684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1c6d5ee-8126-4260-88a3-7920cf4cb532-trusted-ca\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.885303 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.885272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1c6d5ee-8126-4260-88a3-7920cf4cb532-serving-cert\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:16.890817 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:16.890793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prrsw\" (UniqueName: \"kubernetes.io/projected/b1c6d5ee-8126-4260-88a3-7920cf4cb532-kube-api-access-prrsw\") pod \"console-operator-696884fb9c-krl58\" (UID: \"b1c6d5ee-8126-4260-88a3-7920cf4cb532\") " pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:17.039072 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:17.038987    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:17.185811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:17.185754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:17.185811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:17.185815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:17.186086 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:17.185979    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:17.186086 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:17.185982    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.185961613 +0000 UTC m=+79.489978576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:17.186086 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:17.186051    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.186030387 +0000 UTC m=+79.490047335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:17.305024 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:17.304944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-krl58"]
Sep 16 18:46:17.308000 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:17.307969    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c6d5ee_8126_4260_88a3_7920cf4cb532.slice/crio-3758e8b1a2e376fa3a48e4af163372f723f7dd4bfbbaefd623883e9c66a4afa8 WatchSource:0}: Error finding container 3758e8b1a2e376fa3a48e4af163372f723f7dd4bfbbaefd623883e9c66a4afa8: Status 404 returned error can't find the container with id 3758e8b1a2e376fa3a48e4af163372f723f7dd4bfbbaefd623883e9c66a4afa8
Sep 16 18:46:17.568832 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:17.568749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-krl58" event={"ID":"b1c6d5ee-8126-4260-88a3-7920cf4cb532","Type":"ContainerStarted","Data":"3758e8b1a2e376fa3a48e4af163372f723f7dd4bfbbaefd623883e9c66a4afa8"}
Sep 16 18:46:18.196064 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:18.196015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:18.196064 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:18.196066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:18.196282 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:18.196209    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.196184447 +0000 UTC m=+81.500201406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:18.196282 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:18.196213    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:18.196282 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:18.196256    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.19624364 +0000 UTC m=+81.500260599 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:19.575834 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:19.575802    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/0.log"
Sep 16 18:46:19.576231 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:19.575843    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1c6d5ee-8126-4260-88a3-7920cf4cb532" containerID="ac5c2dab69d074cc704762e77ecabfc537c261c67616f38ae27d530fca335bc0" exitCode=255
Sep 16 18:46:19.576231 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:19.575905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-krl58" event={"ID":"b1c6d5ee-8126-4260-88a3-7920cf4cb532","Type":"ContainerDied","Data":"ac5c2dab69d074cc704762e77ecabfc537c261c67616f38ae27d530fca335bc0"}
Sep 16 18:46:19.576231 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:19.576136    2581 scope.go:117] "RemoveContainer" containerID="ac5c2dab69d074cc704762e77ecabfc537c261c67616f38ae27d530fca335bc0"
Sep 16 18:46:20.211931 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.211874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:20.211931 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.211929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:20.212159 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:20.212085    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:24.212062525 +0000 UTC m=+85.516079470 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:20.212159 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:20.212112    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:20.212159 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:20.212153    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:24.212141956 +0000 UTC m=+85.516158904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:20.580900 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.580800    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/1.log"
Sep 16 18:46:20.581254 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.581237    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/0.log"
Sep 16 18:46:20.581303 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.581273    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1c6d5ee-8126-4260-88a3-7920cf4cb532" containerID="0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c" exitCode=255
Sep 16 18:46:20.581354 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.581306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-krl58" event={"ID":"b1c6d5ee-8126-4260-88a3-7920cf4cb532","Type":"ContainerDied","Data":"0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c"}
Sep 16 18:46:20.581354 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.581345    2581 scope.go:117] "RemoveContainer" containerID="ac5c2dab69d074cc704762e77ecabfc537c261c67616f38ae27d530fca335bc0"
Sep 16 18:46:20.581601 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:20.581584    2581 scope.go:117] "RemoveContainer" containerID="0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c"
Sep 16 18:46:20.581777 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:20.581758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-krl58_openshift-console-operator(b1c6d5ee-8126-4260-88a3-7920cf4cb532)\"" pod="openshift-console-operator/console-operator-696884fb9c-krl58" podUID="b1c6d5ee-8126-4260-88a3-7920cf4cb532"
Sep 16 18:46:21.584846 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:21.584819    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/1.log"
Sep 16 18:46:21.585265 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:21.585153    2581 scope.go:117] "RemoveContainer" containerID="0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c"
Sep 16 18:46:21.585315 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:21.585286    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-krl58_openshift-console-operator(b1c6d5ee-8126-4260-88a3-7920cf4cb532)\"" pod="openshift-console-operator/console-operator-696884fb9c-krl58" podUID="b1c6d5ee-8126-4260-88a3-7920cf4cb532"
Sep 16 18:46:22.930049 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.930011    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9sc4w"]
Sep 16 18:46:22.934352 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.934331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:22.936836 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.936813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:46:22.936968 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.936884    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:46:22.937047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.937033    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:46:22.938005 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.937986    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7cpmw\""
Sep 16 18:46:22.938114 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.938098    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:46:22.943241 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:22.943198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9sc4w"]
Sep 16 18:46:23.035270 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.035226    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-crio-socket\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.035270 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.035265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2lg\" (UniqueName: \"kubernetes.io/projected/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-api-access-8w2lg\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.035478 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.035289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.035478 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.035339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-data-volume\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.035478 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.035355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.135868 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.135826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-crio-socket\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.136033 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.135924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2lg\" (UniqueName: \"kubernetes.io/projected/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-api-access-8w2lg\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.136033 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.135957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.136033 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.135989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-data-volume\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.136033 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.135951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-crio-socket\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.136033 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.136004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.136189 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.136064    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.136189 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.136124    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.636109393 +0000 UTC m=+84.940126342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.136189 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.136065    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.136284 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.136203    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:23.636186607 +0000 UTC m=+84.940203552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.136322 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.136306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-data-volume\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.145930 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.145911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2lg\" (UniqueName: \"kubernetes.io/projected/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-api-access-8w2lg\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.638890 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.638826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.639096 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:23.638907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:23.639096 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.638986    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:23.639096 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.638999    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.639096 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.639058    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:24.639043462 +0000 UTC m=+85.943060407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:23.639096 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:23.639071    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:24.639065639 +0000 UTC m=+85.943082584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:24.245084 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:24.245034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:24.245084 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:24.245083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:24.245513 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.245209    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.245188469 +0000 UTC m=+93.549205415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:24.245513 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.245264    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 18:46:24.245513 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.245362    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.245348143 +0000 UTC m=+93.549365103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : secret "router-metrics-certs-default" not found
Sep 16 18:46:24.647945 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:24.647825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:24.647945 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:24.647903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:24.648166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.647983    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:24.648166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.647996    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:24.648166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.648042    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:26.648027614 +0000 UTC m=+87.952044572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:24.648166 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:24.648063    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:26.648050177 +0000 UTC m=+87.952067171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:24.870165 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:24.870135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptxgf_3780632e-fea7-4460-a811-203997052dcb/dns-node-resolver/0.log"
Sep 16 18:46:25.868178 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:25.868154    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67d58_15e98837-18f9-4305-a42e-695792e504f2/node-ca/0.log"
Sep 16 18:46:26.470574 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:26.470547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwz4m_fab74c7b-6b13-4e5e-a494-93793cda1407/serve-healthcheck-canary/0.log"
Sep 16 18:46:26.666668 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:26.666612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:26.666848 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:26.666727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:26.666848 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:26.666738    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:26.666848 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:26.666797    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:30.666781302 +0000 UTC m=+91.970798263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:26.666848 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:26.666824    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:26.667024 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:26.666881    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:30.666851961 +0000 UTC m=+91.970868922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:27.039872 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:27.039810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:27.040262 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:27.039892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:27.040262 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:27.040224    2581 scope.go:117] "RemoveContainer" containerID="0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c"
Sep 16 18:46:27.040374 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:27.040355    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-krl58_openshift-console-operator(b1c6d5ee-8126-4260-88a3-7920cf4cb532)\"" pod="openshift-console-operator/console-operator-696884fb9c-krl58" podUID="b1c6d5ee-8126-4260-88a3-7920cf4cb532"
Sep 16 18:46:30.701287 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:30.701238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:30.701658 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:30.701357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:30.701658 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:30.701365    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:46:30.701658 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:30.701436    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.701419236 +0000 UTC m=+100.005436194 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : configmap "kube-rbac-proxy" not found
Sep 16 18:46:30.701658 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:30.701457    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:30.701658 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:30.701506    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls podName:9c6bdbcf-9c07-47b1-adcd-9a63a682509a nodeName:}" failed. No retries permitted until 2026-09-16 18:46:38.701494597 +0000 UTC m=+100.005511558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9sc4w" (UID: "9c6bdbcf-9c07-47b1-adcd-9a63a682509a") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:32.313066 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:32.313026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:32.313066 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:32.313064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:32.313582 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:32.313203    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle podName:7c6728a2-43fd-4f3e-a9e2-acb7789db2a9 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:48.313184795 +0000 UTC m=+109.617201739 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle") pod "router-default-85465d975b-5frng" (UID: "7c6728a2-43fd-4f3e-a9e2-acb7789db2a9") : configmap references non-existent config key: service-ca.crt
Sep 16 18:46:32.315438 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:32.315414    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-metrics-certs\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:36.448629 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.448601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:46:36.448973 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.448712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:46:36.450741 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.450710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5232b259-8d8e-4a9e-bbec-26720ec51c17-metrics-tls\") pod \"dns-default-pkbvg\" (UID: \"5232b259-8d8e-4a9e-bbec-26720ec51c17\") " pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:46:36.450841 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.450782    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"image-registry-89845797-2jj42\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") " pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:46:36.457560 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.457542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vf4dn\""
Sep 16 18:46:36.465660 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.465643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:46:36.735000 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.734910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pkbvg"]
Sep 16 18:46:36.738106 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.738086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bk56d\""
Sep 16 18:46:36.739632 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:36.739608    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5232b259_8d8e_4a9e_bbec_26720ec51c17.slice/crio-53353105f986b606ef640c70a8a089ea55bd66560e1757eb46d8a1125dfcbc15 WatchSource:0}: Error finding container 53353105f986b606ef640c70a8a089ea55bd66560e1757eb46d8a1125dfcbc15: Status 404 returned error can't find the container with id 53353105f986b606ef640c70a8a089ea55bd66560e1757eb46d8a1125dfcbc15
Sep 16 18:46:36.746004 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:36.745985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:46:37.021320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:37.021240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-89845797-2jj42"]
Sep 16 18:46:37.024268 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:37.024243    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b4ee46_93f9_4f8f_9bf2_d83b3e3448a2.slice/crio-9219e8c15207909f843f5fe62e5c94d909324009a6b0cc91eb5089a92cb6a645 WatchSource:0}: Error finding container 9219e8c15207909f843f5fe62e5c94d909324009a6b0cc91eb5089a92cb6a645: Status 404 returned error can't find the container with id 9219e8c15207909f843f5fe62e5c94d909324009a6b0cc91eb5089a92cb6a645
Sep 16 18:46:37.626792 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:37.626753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkbvg" event={"ID":"5232b259-8d8e-4a9e-bbec-26720ec51c17","Type":"ContainerStarted","Data":"53353105f986b606ef640c70a8a089ea55bd66560e1757eb46d8a1125dfcbc15"}
Sep 16 18:46:37.628645 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:37.628612    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89845797-2jj42" event={"ID":"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2","Type":"ContainerStarted","Data":"3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af"}
Sep 16 18:46:37.628774 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:37.628661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89845797-2jj42" event={"ID":"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2","Type":"ContainerStarted","Data":"9219e8c15207909f843f5fe62e5c94d909324009a6b0cc91eb5089a92cb6a645"}
Sep 16 18:46:37.628774 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:37.628720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:46:37.651060 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:37.651006    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-89845797-2jj42" podStartSLOduration=98.650991778 podStartE2EDuration="1m38.650991778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:37.649579791 +0000 UTC m=+98.953596757" watchObservedRunningTime="2026-09-16 18:46:37.650991778 +0000 UTC m=+98.955008744"
Sep 16 18:46:38.634324 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.634231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkbvg" event={"ID":"5232b259-8d8e-4a9e-bbec-26720ec51c17","Type":"ContainerStarted","Data":"ac6b64c790d68aca22b08f1aaddc4d571e9b60c1aafa5e5a545064a35a509d24"}
Sep 16 18:46:38.634324 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.634306    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pkbvg" event={"ID":"5232b259-8d8e-4a9e-bbec-26720ec51c17","Type":"ContainerStarted","Data":"793c449629b2f7d13caf91f11ae7e09fc2e5c3db2305b72f6a6641723a7ba124"}
Sep 16 18:46:38.652790 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.652749    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pkbvg" podStartSLOduration=65.180397594 podStartE2EDuration="1m6.652736741s" totalImagesPullingTime="1.472339147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:38.652647083 +0000 UTC m=+99.956664049" watchObservedRunningTime="2026-09-16 18:46:38.652736741 +0000 UTC m=+99.956753708"
Sep 16 18:46:38.766324 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.766296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:38.766702 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.766639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:38.766888 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.766841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:38.769064 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.769043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c6bdbcf-9c07-47b1-adcd-9a63a682509a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9sc4w\" (UID: \"9c6bdbcf-9c07-47b1-adcd-9a63a682509a\") " pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:38.843080 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:38.843052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9sc4w"
Sep 16 18:46:39.114639 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:39.114605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9sc4w"]
Sep 16 18:46:39.117283 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:39.117259    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6bdbcf_9c07_47b1_adcd_9a63a682509a.slice/crio-41d63d22e3f12132360d6972962cae2a84a781b7b35481acff2874df866575c0 WatchSource:0}: Error finding container 41d63d22e3f12132360d6972962cae2a84a781b7b35481acff2874df866575c0: Status 404 returned error can't find the container with id 41d63d22e3f12132360d6972962cae2a84a781b7b35481acff2874df866575c0
Sep 16 18:46:39.639088 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:39.639054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9sc4w" event={"ID":"9c6bdbcf-9c07-47b1-adcd-9a63a682509a","Type":"ContainerStarted","Data":"2cf9b993a61242e2838098d24fa73a9befe3c541e88a8190a8ea78f8d834da0d"}
Sep 16 18:46:39.639544 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:39.639112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9sc4w" event={"ID":"9c6bdbcf-9c07-47b1-adcd-9a63a682509a","Type":"ContainerStarted","Data":"41d63d22e3f12132360d6972962cae2a84a781b7b35481acff2874df866575c0"}
Sep 16 18:46:39.639544 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:39.639307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:46:40.216438 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.216412    2581 scope.go:117] "RemoveContainer" containerID="0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c"
Sep 16 18:46:40.553953 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.553849    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qb9wt"
Sep 16 18:46:40.645609 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.645570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9sc4w" event={"ID":"9c6bdbcf-9c07-47b1-adcd-9a63a682509a","Type":"ContainerStarted","Data":"ba6851b561a9e1f7008e0ece7cb77c92f14350c41d3204dec75fa07c2e4b99bc"}
Sep 16 18:46:40.647673 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.647589    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:46:40.649505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.649458    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/1.log"
Sep 16 18:46:40.649637 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.649520    2581 generic.go:356] "Generic (PLEG): container finished" podID="b1c6d5ee-8126-4260-88a3-7920cf4cb532" containerID="4ea61f814d175863f9aafdc13f737f16775a550b7f0b7534b966b350468edf91" exitCode=255
Sep 16 18:46:40.649637 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.649608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-krl58" event={"ID":"b1c6d5ee-8126-4260-88a3-7920cf4cb532","Type":"ContainerDied","Data":"4ea61f814d175863f9aafdc13f737f16775a550b7f0b7534b966b350468edf91"}
Sep 16 18:46:40.649750 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.649672    2581 scope.go:117] "RemoveContainer" containerID="0470d003853aeecc7734d26085d0389e8533813275744a0c813df92f1f17605c"
Sep 16 18:46:40.650299 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:40.650133    2581 scope.go:117] "RemoveContainer" containerID="4ea61f814d175863f9aafdc13f737f16775a550b7f0b7534b966b350468edf91"
Sep 16 18:46:40.650425 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:40.650299    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-krl58_openshift-console-operator(b1c6d5ee-8126-4260-88a3-7920cf4cb532)\"" pod="openshift-console-operator/console-operator-696884fb9c-krl58" podUID="b1c6d5ee-8126-4260-88a3-7920cf4cb532"
Sep 16 18:46:41.654810 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:41.654786    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:46:42.661670 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:42.661636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9sc4w" event={"ID":"9c6bdbcf-9c07-47b1-adcd-9a63a682509a","Type":"ContainerStarted","Data":"d0c2881d71e2792e00d8a80c28fabf51efa819c9e9551c154540f603fabaf371"}
Sep 16 18:46:42.681928 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:42.681871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9sc4w" podStartSLOduration=18.24296636 podStartE2EDuration="20.681840615s" totalImagesPullingTime="2.438874255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:42.680495123 +0000 UTC m=+103.984512133" watchObservedRunningTime="2026-09-16 18:46:42.681840615 +0000 UTC m=+103.985857582"
Sep 16 18:46:47.039326 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:47.039277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:47.039796 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:47.039348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:46:47.039796 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:47.039620    2581 scope.go:117] "RemoveContainer" containerID="4ea61f814d175863f9aafdc13f737f16775a550b7f0b7534b966b350468edf91"
Sep 16 18:46:47.039796 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:46:47.039764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-krl58_openshift-console-operator(b1c6d5ee-8126-4260-88a3-7920cf4cb532)\"" pod="openshift-console-operator/console-operator-696884fb9c-krl58" podUID="b1c6d5ee-8126-4260-88a3-7920cf4cb532"
Sep 16 18:46:48.334253 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.334214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:48.334813 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.334794    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6728a2-43fd-4f3e-a9e2-acb7789db2a9-service-ca-bundle\") pod \"router-default-85465d975b-5frng\" (UID: \"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9\") " pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:48.343345 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.343325    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:48.612085 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.611988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85465d975b-5frng"]
Sep 16 18:46:48.614832 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:48.614807    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6728a2_43fd_4f3e_a9e2_acb7789db2a9.slice/crio-d105a401f14e1fd117c65423cda6d019f7882a787347f60a3e77f3a287e96427 WatchSource:0}: Error finding container d105a401f14e1fd117c65423cda6d019f7882a787347f60a3e77f3a287e96427: Status 404 returned error can't find the container with id d105a401f14e1fd117c65423cda6d019f7882a787347f60a3e77f3a287e96427
Sep 16 18:46:48.681386 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.681357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85465d975b-5frng" event={"ID":"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9","Type":"ContainerStarted","Data":"7005def084062931f75c05e486499afba9a62e7b7a3bd5830220296d60ad9268"}
Sep 16 18:46:48.681617 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.681412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85465d975b-5frng" event={"ID":"7c6728a2-43fd-4f3e-a9e2-acb7789db2a9","Type":"ContainerStarted","Data":"d105a401f14e1fd117c65423cda6d019f7882a787347f60a3e77f3a287e96427"}
Sep 16 18:46:48.700311 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:48.700271    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-85465d975b-5frng" podStartSLOduration=32.700255702 podStartE2EDuration="32.700255702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:48.699481852 +0000 UTC m=+110.003498818" watchObservedRunningTime="2026-09-16 18:46:48.700255702 +0000 UTC m=+110.004272668"
Sep 16 18:46:49.343490 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:49.343461    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:49.347222 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:49.347201    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:49.652329 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:49.652301    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pkbvg"
Sep 16 18:46:49.684590 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:49.684564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:49.685699 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:49.685679    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85465d975b-5frng"
Sep 16 18:46:50.049020 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.048987    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"]
Sep 16 18:46:50.054695 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.054675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.057901 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.057879    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 18:46:50.058019 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.057902    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 18:46:50.058019 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.057881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 18:46:50.058181 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.058165    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 18:46:50.058282 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.058263    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 18:46:50.058657 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.058640    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 18:46:50.058967 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.058949    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 18:46:50.070984 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.070966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"]
Sep 16 18:46:50.149301 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.149273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-ca\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.149301 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.149305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.149522 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.149324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.149522 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.149388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-hub\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.149522 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.149451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7d95\" (UniqueName: \"kubernetes.io/projected/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-kube-api-access-q7d95\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.149644 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.149530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.249980 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.249951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.249980 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.249982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-ca\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.250190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.250004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.250190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.250021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.250190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.250036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-hub\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.250190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.250077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7d95\" (UniqueName: \"kubernetes.io/projected/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-kube-api-access-q7d95\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.250776 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.250745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.252482 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.252451    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-ca\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.252665 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.252644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.252757 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.252737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-hub\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.252820 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.252798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.265887 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.265821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7d95\" (UniqueName: \"kubernetes.io/projected/fa25bbbe-c34e-4e20-be9c-d6525e84a98a-kube-api-access-q7d95\") pod \"cluster-proxy-proxy-agent-84c7db4c46-xnq4d\" (UID: \"fa25bbbe-c34e-4e20-be9c-d6525e84a98a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.374300 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.374260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"
Sep 16 18:46:50.641658 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.641590    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d"]
Sep 16 18:46:50.644498 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:50.644470    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa25bbbe_c34e_4e20_be9c_d6525e84a98a.slice/crio-97385c6f64aff0f3d210b1844852a09764fa6fb3f8d753405030e507ec752687 WatchSource:0}: Error finding container 97385c6f64aff0f3d210b1844852a09764fa6fb3f8d753405030e507ec752687: Status 404 returned error can't find the container with id 97385c6f64aff0f3d210b1844852a09764fa6fb3f8d753405030e507ec752687
Sep 16 18:46:50.688347 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:50.688317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d" event={"ID":"fa25bbbe-c34e-4e20-be9c-d6525e84a98a","Type":"ContainerStarted","Data":"97385c6f64aff0f3d210b1844852a09764fa6fb3f8d753405030e507ec752687"}
Sep 16 18:46:54.705868 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:54.705817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d" event={"ID":"fa25bbbe-c34e-4e20-be9c-d6525e84a98a","Type":"ContainerStarted","Data":"c7f4a9451d22beae9fd085528c4c7f133dbbeef20649eb2c7f2057d0465cd7af"}
Sep 16 18:46:54.706223 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:54.705884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d" event={"ID":"fa25bbbe-c34e-4e20-be9c-d6525e84a98a","Type":"ContainerStarted","Data":"0acaa44b3b4f22eee2356d2b008a1def448d50f3ad9186f705564322482b7fce"}
Sep 16 18:46:54.706223 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:54.705899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d" event={"ID":"fa25bbbe-c34e-4e20-be9c-d6525e84a98a","Type":"ContainerStarted","Data":"b9d0567d164001e64cf31c132651654a30894cef98be4eda3764bf719b16a2e3"}
Sep 16 18:46:54.725073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:54.725026    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84c7db4c46-xnq4d" podStartSLOduration=1.082897453 podStartE2EDuration="4.725012878s" totalImagesPullingTime="3.642115425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:54.72424629 +0000 UTC m=+116.028263261" watchObservedRunningTime="2026-09-16 18:46:54.725012878 +0000 UTC m=+116.029029878"
Sep 16 18:46:56.750875 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:56.750826    2581 patch_prober.go:28] interesting pod/image-registry-89845797-2jj42 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:46:56.751299 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:56.750896    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-89845797-2jj42" podUID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:46:58.638751 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.638720    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:46:58.711784 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.711743    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mzvlr"]
Sep 16 18:46:58.716177 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.716152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.718615 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.718593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kdm59\""
Sep 16 18:46:58.718972 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.718953    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:58.719079 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.719026    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:58.719219 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.719202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:46:58.719219 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.719208    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:46:58.719577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.719558    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:58.719877 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.719842    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:58.814666 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-tls\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.814666 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-root\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.814844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5c5d\" (UniqueName: \"kubernetes.io/projected/4bf8bb78-8dce-4a5d-84fa-803067c5944c-kube-api-access-q5c5d\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.814844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.814844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-wtmp\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.814844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-textfile\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.814844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-sys\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.815023 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bf8bb78-8dce-4a5d-84fa-803067c5944c-metrics-client-ca\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.815023 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.814950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-wtmp\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-textfile\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-sys\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bf8bb78-8dce-4a5d-84fa-803067c5944c-metrics-client-ca\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.915815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-tls\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916134 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-sys\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916134 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-root\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916134 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915941    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-wtmp\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916134 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4bf8bb78-8dce-4a5d-84fa-803067c5944c-root\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916134 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.915998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5c5d\" (UniqueName: \"kubernetes.io/projected/4bf8bb78-8dce-4a5d-84fa-803067c5944c-kube-api-access-q5c5d\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916400 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.916382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916730 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.916710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-textfile\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.916952 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.916936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bf8bb78-8dce-4a5d-84fa-803067c5944c-metrics-client-ca\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.918096 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.918080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-tls\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.918186 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.918128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4bf8bb78-8dce-4a5d-84fa-803067c5944c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:58.934014 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:58.933988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5c5d\" (UniqueName: \"kubernetes.io/projected/4bf8bb78-8dce-4a5d-84fa-803067c5944c-kube-api-access-q5c5d\") pod \"node-exporter-mzvlr\" (UID: \"4bf8bb78-8dce-4a5d-84fa-803067c5944c\") " pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:59.027851 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:59.027821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mzvlr"
Sep 16 18:46:59.039210 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:46:59.039187    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf8bb78_8dce_4a5d_84fa_803067c5944c.slice/crio-5d27811960ec71779ef24baf2557811ad87d8652e52e9e10295cfda12ecc8794 WatchSource:0}: Error finding container 5d27811960ec71779ef24baf2557811ad87d8652e52e9e10295cfda12ecc8794: Status 404 returned error can't find the container with id 5d27811960ec71779ef24baf2557811ad87d8652e52e9e10295cfda12ecc8794
Sep 16 18:46:59.723113 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:46:59.723076    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvlr" event={"ID":"4bf8bb78-8dce-4a5d-84fa-803067c5944c","Type":"ContainerStarted","Data":"5d27811960ec71779ef24baf2557811ad87d8652e52e9e10295cfda12ecc8794"}
Sep 16 18:47:00.727750 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:00.727714    2581 generic.go:356] "Generic (PLEG): container finished" podID="4bf8bb78-8dce-4a5d-84fa-803067c5944c" containerID="807db10e984c957d5705a23da5dad4de4efdec5b1892c78bd264fd45905d98c9" exitCode=0
Sep 16 18:47:00.728164 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:00.727770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvlr" event={"ID":"4bf8bb78-8dce-4a5d-84fa-803067c5944c","Type":"ContainerDied","Data":"807db10e984c957d5705a23da5dad4de4efdec5b1892c78bd264fd45905d98c9"}
Sep 16 18:47:01.217131 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.217095    2581 scope.go:117] "RemoveContainer" containerID="4ea61f814d175863f9aafdc13f737f16775a550b7f0b7534b966b350468edf91"
Sep 16 18:47:01.732375 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.732343    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:47:01.732810 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.732457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-krl58" event={"ID":"b1c6d5ee-8126-4260-88a3-7920cf4cb532","Type":"ContainerStarted","Data":"71055402a51b25dd4301b1390464f4ef2f9e1c5c314bae103af78294a576473c"}
Sep 16 18:47:01.732810 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.732747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:47:01.734794 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.734769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvlr" event={"ID":"4bf8bb78-8dce-4a5d-84fa-803067c5944c","Type":"ContainerStarted","Data":"adcd0a14ca724fb578c82c2125c13f65b8b1b9031da4fedbe9a87e44bf2bb734"}
Sep 16 18:47:01.734928 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.734810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mzvlr" event={"ID":"4bf8bb78-8dce-4a5d-84fa-803067c5944c","Type":"ContainerStarted","Data":"b654245543451be989267224ed5cdec3d27c09511f3d043542376534626f6889"}
Sep 16 18:47:01.752931 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.752888    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-krl58" podStartSLOduration=44.107551278 podStartE2EDuration="45.752852547s" totalImagesPullingTime="1.645301269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.752613889 +0000 UTC m=+123.056630857" watchObservedRunningTime="2026-09-16 18:47:01.752852547 +0000 UTC m=+123.056869514"
Sep 16 18:47:01.773921 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:01.773884    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mzvlr" podStartSLOduration=2.703387199 podStartE2EDuration="3.773872821s" totalImagesPullingTime="1.010774474s" totalInitContainerRuntime="59.711148ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:01.773019355 +0000 UTC m=+123.077036338" watchObservedRunningTime="2026-09-16 18:47:01.773872821 +0000 UTC m=+123.077889785"
Sep 16 18:47:02.016998 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:02.016928    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-krl58"
Sep 16 18:47:08.086969 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:08.086927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:47:08.089161 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:08.089142    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d082ca-6663-4b4a-9dbb-68e149d6fc10-metrics-certs\") pod \"network-metrics-daemon-tgznf\" (UID: \"77d082ca-6663-4b4a-9dbb-68e149d6fc10\") " pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:47:08.138634 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:08.138609    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m77pk\""
Sep 16 18:47:08.146701 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:08.146680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tgznf"
Sep 16 18:47:08.415068 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:08.414917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tgznf"]
Sep 16 18:47:08.417457 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:47:08.417426    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d082ca_6663_4b4a_9dbb_68e149d6fc10.slice/crio-a0a81b0fce490b30e8be9128a6637c6f52fb545a8b073e7ed922d40d037142e4 WatchSource:0}: Error finding container a0a81b0fce490b30e8be9128a6637c6f52fb545a8b073e7ed922d40d037142e4: Status 404 returned error can't find the container with id a0a81b0fce490b30e8be9128a6637c6f52fb545a8b073e7ed922d40d037142e4
Sep 16 18:47:08.756596 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:08.756521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgznf" event={"ID":"77d082ca-6663-4b4a-9dbb-68e149d6fc10","Type":"ContainerStarted","Data":"a0a81b0fce490b30e8be9128a6637c6f52fb545a8b073e7ed922d40d037142e4"}
Sep 16 18:47:09.761190 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:09.761154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgznf" event={"ID":"77d082ca-6663-4b4a-9dbb-68e149d6fc10","Type":"ContainerStarted","Data":"cbafd8efa279800b0c8a65715aa044eb014c584ac7fe3c73c2ba29da3cf750aa"}
Sep 16 18:47:09.761626 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:09.761230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tgznf" event={"ID":"77d082ca-6663-4b4a-9dbb-68e149d6fc10","Type":"ContainerStarted","Data":"548496e4a75b8e1af43da227f6b5e22cdc5ad3b7f12462e8a83f67f25c93f52b"}
Sep 16 18:47:09.778976 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:09.778928    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tgznf" podStartSLOduration=129.946085565 podStartE2EDuration="2m10.77891508s" totalImagesPullingTime="832.829515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:09.777511872 +0000 UTC m=+131.081528874" watchObservedRunningTime="2026-09-16 18:47:09.77891508 +0000 UTC m=+131.082932047"
Sep 16 18:47:13.207902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:13.207869    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-89845797-2jj42"]
Sep 16 18:47:38.226688 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.226648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-89845797-2jj42" podUID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" containerName="registry" containerID="cri-o://3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af" gracePeriod=30
Sep 16 18:47:38.458421 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.458395    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:47:38.492444 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492386    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-bound-sa-token\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492444 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-ca-trust-extracted\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-installation-pull-secrets\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scgsm\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-kube-api-access-scgsm\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492730 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492660    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-image-registry-private-configuration\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492730 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492697    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492828 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492736    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-trusted-ca\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.492828 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.492768    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-certificates\") pod \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\" (UID: \"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2\") "
Sep 16 18:47:38.493422 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.493395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:38.493620 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.493570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:38.495320 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.495262    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:38.495423 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.495336    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:38.495423 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.495368    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:38.495683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.495663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-kube-api-access-scgsm" (OuterVolumeSpecName: "kube-api-access-scgsm") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "kube-api-access-scgsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:38.495803 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.495748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:38.502177 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.502150    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" (UID: "e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:38.594109 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594088    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-installation-pull-secrets\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594109 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594106    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scgsm\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-kube-api-access-scgsm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594117    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-image-registry-private-configuration\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594127    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-tls\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594135    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-trusted-ca\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594142    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-registry-certificates\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594150    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-bound-sa-token\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.594201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.594158    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2-ca-trust-extracted\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:47:38.861812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.861721    2581 generic.go:356] "Generic (PLEG): container finished" podID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" containerID="3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af" exitCode=0
Sep 16 18:47:38.861812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.861779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89845797-2jj42" event={"ID":"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2","Type":"ContainerDied","Data":"3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af"}
Sep 16 18:47:38.861812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.861782    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-89845797-2jj42"
Sep 16 18:47:38.862082 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.861839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-89845797-2jj42" event={"ID":"e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2","Type":"ContainerDied","Data":"9219e8c15207909f843f5fe62e5c94d909324009a6b0cc91eb5089a92cb6a645"}
Sep 16 18:47:38.862082 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.861875    2581 scope.go:117] "RemoveContainer" containerID="3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af"
Sep 16 18:47:38.870985 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.870968    2581 scope.go:117] "RemoveContainer" containerID="3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af"
Sep 16 18:47:38.871260 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:47:38.871239    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af\": container with ID starting with 3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af not found: ID does not exist" containerID="3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af"
Sep 16 18:47:38.871325 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.871266    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af"} err="failed to get container status \"3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af\": rpc error: code = NotFound desc = could not find container \"3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af\": container with ID starting with 3f55535475b9dacffb799f0db31a342cc7111893d1e9605dd10e21884edbe1af not found: ID does not exist"
Sep 16 18:47:38.884678 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.884653    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-89845797-2jj42"]
Sep 16 18:47:38.889716 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:38.889684    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-89845797-2jj42"]
Sep 16 18:47:39.219614 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:47:39.219585    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" path="/var/lib/kubelet/pods/e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2/volumes"
Sep 16 18:49:59.109124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:49:59.109096    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:49:59.112122 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:49:59.112101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:49:59.113826 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:49:59.113804    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:49:59.116694 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:49:59.116675    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:49:59.118936 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:49:59.118916    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:50:15.726211 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.726179    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"]
Sep 16 18:50:15.728466 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.726397    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" containerName="registry"
Sep 16 18:50:15.728466 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.726408    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" containerName="registry"
Sep 16 18:50:15.728466 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.726456    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b4ee46-93f9-4f8f-9bf2-d83b3e3448a2" containerName="registry"
Sep 16 18:50:15.729284 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.729269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.732058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.732034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vp8rv\""
Sep 16 18:50:15.732192 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.732116    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:50:15.733262 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.733241    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:50:15.741551 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.741532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"]
Sep 16 18:50:15.832849 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.832819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.833010 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.832877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwr7\" (UniqueName: \"kubernetes.io/projected/15e7aff4-b056-4aca-a0ec-5c789183e6c5-kube-api-access-vwwr7\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.833010 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.832926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.934128 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.934093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.934279 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.934138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwr7\" (UniqueName: \"kubernetes.io/projected/15e7aff4-b056-4aca-a0ec-5c789183e6c5-kube-api-access-vwwr7\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.934279 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.934162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.934490 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.934458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.934490 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.934485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:15.942526 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:15.942502    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwr7\" (UniqueName: \"kubernetes.io/projected/15e7aff4-b056-4aca-a0ec-5c789183e6c5-kube-api-access-vwwr7\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:16.038344 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:16.038277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:16.306809 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:16.306737    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"]
Sep 16 18:50:16.308947 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:50:16.308908    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e7aff4_b056_4aca_a0ec_5c789183e6c5.slice/crio-a8aabf47b5e5c6b22fa96930920d466f0d8c163daf473d079c632a57e03b4566 WatchSource:0}: Error finding container a8aabf47b5e5c6b22fa96930920d466f0d8c163daf473d079c632a57e03b4566: Status 404 returned error can't find the container with id a8aabf47b5e5c6b22fa96930920d466f0d8c163daf473d079c632a57e03b4566
Sep 16 18:50:16.310765 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:16.310747    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:50:16.365865 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:16.365826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45" event={"ID":"15e7aff4-b056-4aca-a0ec-5c789183e6c5","Type":"ContainerStarted","Data":"a8aabf47b5e5c6b22fa96930920d466f0d8c163daf473d079c632a57e03b4566"}
Sep 16 18:50:22.390442 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:22.390357    2581 generic.go:356] "Generic (PLEG): container finished" podID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerID="a2b1071d2f0b238b4b437c3dd1aefc93a5599583184f78a2aca2dad38f1b1679" exitCode=0
Sep 16 18:50:22.390442 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:22.390418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45" event={"ID":"15e7aff4-b056-4aca-a0ec-5c789183e6c5","Type":"ContainerDied","Data":"a2b1071d2f0b238b4b437c3dd1aefc93a5599583184f78a2aca2dad38f1b1679"}
Sep 16 18:50:29.413632 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:29.413593    2581 generic.go:356] "Generic (PLEG): container finished" podID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerID="6786bd0ed0a2b04ff61f4824725568e19cafb33893077a2bac471e4febb905ad" exitCode=0
Sep 16 18:50:29.414124 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:29.413651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45" event={"ID":"15e7aff4-b056-4aca-a0ec-5c789183e6c5","Type":"ContainerDied","Data":"6786bd0ed0a2b04ff61f4824725568e19cafb33893077a2bac471e4febb905ad"}
Sep 16 18:50:36.440385 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:36.440350    2581 generic.go:356] "Generic (PLEG): container finished" podID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerID="5219f7cc84f6194a19f1e20e15ce6b09f10471d4742cff8e6a598f2983319cb8" exitCode=0
Sep 16 18:50:36.440707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:36.440397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45" event={"ID":"15e7aff4-b056-4aca-a0ec-5c789183e6c5","Type":"ContainerDied","Data":"5219f7cc84f6194a19f1e20e15ce6b09f10471d4742cff8e6a598f2983319cb8"}
Sep 16 18:50:37.565785 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.565760    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:37.594426 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.594388    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-util\") pod \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") "
Sep 16 18:50:37.594585 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.594430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwwr7\" (UniqueName: \"kubernetes.io/projected/15e7aff4-b056-4aca-a0ec-5c789183e6c5-kube-api-access-vwwr7\") pod \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") "
Sep 16 18:50:37.594585 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.594469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-bundle\") pod \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\" (UID: \"15e7aff4-b056-4aca-a0ec-5c789183e6c5\") "
Sep 16 18:50:37.595168 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.595141    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-bundle" (OuterVolumeSpecName: "bundle") pod "15e7aff4-b056-4aca-a0ec-5c789183e6c5" (UID: "15e7aff4-b056-4aca-a0ec-5c789183e6c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:37.596537 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.596515    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e7aff4-b056-4aca-a0ec-5c789183e6c5-kube-api-access-vwwr7" (OuterVolumeSpecName: "kube-api-access-vwwr7") pod "15e7aff4-b056-4aca-a0ec-5c789183e6c5" (UID: "15e7aff4-b056-4aca-a0ec-5c789183e6c5"). InnerVolumeSpecName "kube-api-access-vwwr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:37.601724 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.601702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-util" (OuterVolumeSpecName: "util") pod "15e7aff4-b056-4aca-a0ec-5c789183e6c5" (UID: "15e7aff4-b056-4aca-a0ec-5c789183e6c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:50:37.694956 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.694886    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:50:37.694956 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.694912    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwwr7\" (UniqueName: \"kubernetes.io/projected/15e7aff4-b056-4aca-a0ec-5c789183e6c5-kube-api-access-vwwr7\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:50:37.694956 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:37.694922    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/15e7aff4-b056-4aca-a0ec-5c789183e6c5-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:50:38.448597 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:38.448563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45" event={"ID":"15e7aff4-b056-4aca-a0ec-5c789183e6c5","Type":"ContainerDied","Data":"a8aabf47b5e5c6b22fa96930920d466f0d8c163daf473d079c632a57e03b4566"}
Sep 16 18:50:38.448768 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:38.448613    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8aabf47b5e5c6b22fa96930920d466f0d8c163daf473d079c632a57e03b4566"
Sep 16 18:50:38.448768 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:38.448573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1al7s45"
Sep 16 18:50:47.913390 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hk4b6"]
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913675    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="util"
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913693    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="util"
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913712    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="extract"
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913720    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="extract"
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913734    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="pull"
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913741    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="pull"
Sep 16 18:50:47.914057 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.913801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e7aff4-b056-4aca-a0ec-5c789183e6c5" containerName="extract"
Sep 16 18:50:47.915819 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.915799    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:47.919051 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.919028    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:50:47.920237 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.920203    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:50:47.920237 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.920208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:50:47.920399 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.920240    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5wkns\""
Sep 16 18:50:47.920399 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.920252    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:50:47.920399 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.920280    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:50:47.928394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.928364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hk4b6"]
Sep 16 18:50:47.966931 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.966883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:47.966931 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.966931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-cabundle0\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:47.967141 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:47.967031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6bc\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-kube-api-access-6j6bc\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.067996 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.067955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.067996 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.067998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-cabundle0\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.068201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.068058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6bc\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-kube-api-access-6j6bc\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.068201 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.068116    2581 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:50:48.068201 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.068134    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:50:48.068201 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.068141    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:50:48.068201 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.068153    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-hk4b6: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:50:48.068403 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.068239    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates podName:3ca16888-6858-4eb3-b9ff-63f23ec6f8fa nodeName:}" failed. No retries permitted until 2026-09-16 18:50:48.568223889 +0000 UTC m=+349.872240858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates") pod "keda-operator-579c8958f7-hk4b6" (UID: "3ca16888-6858-4eb3-b9ff-63f23ec6f8fa") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:50:48.068651 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.068628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-cabundle0\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.076733 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.076700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6bc\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-kube-api-access-6j6bc\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.552497 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.552459    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"]
Sep 16 18:50:48.554666 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.554643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.557137 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.557118    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:50:48.564434 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.564400    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"]
Sep 16 18:50:48.572109 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.572087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4bbe122e-7269-44c8-8c65-215c5dcac3ad-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.572239 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.572150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzhz\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-kube-api-access-6jzhz\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.572239 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.572194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:48.572345 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.572252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.572345 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.572329    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:50:48.572440 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.572346    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:50:48.572440 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.572358    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-hk4b6: references non-existent secret key: ca.crt
Sep 16 18:50:48.572440 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.572427    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates podName:3ca16888-6858-4eb3-b9ff-63f23ec6f8fa nodeName:}" failed. No retries permitted until 2026-09-16 18:50:49.572408009 +0000 UTC m=+350.876424969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates") pod "keda-operator-579c8958f7-hk4b6" (UID: "3ca16888-6858-4eb3-b9ff-63f23ec6f8fa") : references non-existent secret key: ca.crt
Sep 16 18:50:48.673400 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.673360    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.673603 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.673419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4bbe122e-7269-44c8-8c65-215c5dcac3ad-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.673603 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.673468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzhz\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-kube-api-access-6jzhz\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.673603 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.673503    2581 secret.go:281] references non-existent secret key: tls.crt
Sep 16 18:50:48.673603 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.673527    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 16 18:50:48.673603 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.673544    2581 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 16 18:50:48.673603 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.673560    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 16 18:50:48.673830 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:48.673611    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-certificates podName:4bbe122e-7269-44c8-8c65-215c5dcac3ad nodeName:}" failed. No retries permitted until 2026-09-16 18:50:49.173597915 +0000 UTC m=+350.477614864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-certificates") pod "keda-metrics-apiserver-77857b7b98-nlm6t" (UID: "4bbe122e-7269-44c8-8c65-215c5dcac3ad") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Sep 16 18:50:48.673909 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.673896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/4bbe122e-7269-44c8-8c65-215c5dcac3ad-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:48.682574 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:48.682553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzhz\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-kube-api-access-6jzhz\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:49.037355 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.037322    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-q6qrt"]
Sep 16 18:50:49.039305 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.039287    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.042283 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.042266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:50:49.053728 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.053704    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-q6qrt"]
Sep 16 18:50:49.076963 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.076929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-certificates\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.077110 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.076986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnqdt\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-kube-api-access-mnqdt\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.177713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.177672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:49.177713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.177715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-certificates\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.177933 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.177827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnqdt\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-kube-api-access-mnqdt\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.177970 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:49.177955    2581 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 18:50:49.178002 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:49.177980    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-q6qrt: secret "keda-admission-webhooks-certs" not found
Sep 16 18:50:49.178050 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:50:49.178040    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-certificates podName:7d1be23c-0efb-474a-af87-6506d4f54984 nodeName:}" failed. No retries permitted until 2026-09-16 18:50:49.678020238 +0000 UTC m=+350.982037187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-certificates") pod "keda-admission-6c4d864b86-q6qrt" (UID: "7d1be23c-0efb-474a-af87-6506d4f54984") : secret "keda-admission-webhooks-certs" not found
Sep 16 18:50:49.180189 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.180158    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bbe122e-7269-44c8-8c65-215c5dcac3ad-certificates\") pod \"keda-metrics-apiserver-77857b7b98-nlm6t\" (UID: \"4bbe122e-7269-44c8-8c65-215c5dcac3ad\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:49.187191 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.187163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnqdt\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-kube-api-access-mnqdt\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.464957 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.464922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:49.581726 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.581681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:49.584225 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.584199    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/3ca16888-6858-4eb3-b9ff-63f23ec6f8fa-certificates\") pod \"keda-operator-579c8958f7-hk4b6\" (UID: \"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa\") " pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:49.682361 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.682331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-certificates\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.684841 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.684814    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d1be23c-0efb-474a-af87-6506d4f54984-certificates\") pod \"keda-admission-6c4d864b86-q6qrt\" (UID: \"7d1be23c-0efb-474a-af87-6506d4f54984\") " pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:49.725906 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.725809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:49.735135 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.734973    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"]
Sep 16 18:50:49.737671 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:50:49.737643    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbe122e_7269_44c8_8c65_215c5dcac3ad.slice/crio-565102f527339b706e84068c924ac191140ac5cf1ce8353fb84c8f76652dadc7 WatchSource:0}: Error finding container 565102f527339b706e84068c924ac191140ac5cf1ce8353fb84c8f76652dadc7: Status 404 returned error can't find the container with id 565102f527339b706e84068c924ac191140ac5cf1ce8353fb84c8f76652dadc7
Sep 16 18:50:49.948773 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:49.948716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:50.001118 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:50.001089    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-hk4b6"]
Sep 16 18:50:50.005723 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:50:50.005695    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca16888_6858_4eb3_b9ff_63f23ec6f8fa.slice/crio-ef9c498992851199e70cc00d9138c70b56cce080a7646545443ea604ef8ed093 WatchSource:0}: Error finding container ef9c498992851199e70cc00d9138c70b56cce080a7646545443ea604ef8ed093: Status 404 returned error can't find the container with id ef9c498992851199e70cc00d9138c70b56cce080a7646545443ea604ef8ed093
Sep 16 18:50:50.225131 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:50.225040    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-q6qrt"]
Sep 16 18:50:50.227605 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:50:50.227567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1be23c_0efb_474a_af87_6506d4f54984.slice/crio-56a2df73493b6dd6697b1dec65f4ae9ad0744a969e4e54362a2837a56ead3d77 WatchSource:0}: Error finding container 56a2df73493b6dd6697b1dec65f4ae9ad0744a969e4e54362a2837a56ead3d77: Status 404 returned error can't find the container with id 56a2df73493b6dd6697b1dec65f4ae9ad0744a969e4e54362a2837a56ead3d77
Sep 16 18:50:50.492528 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:50.492487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt" event={"ID":"7d1be23c-0efb-474a-af87-6506d4f54984","Type":"ContainerStarted","Data":"56a2df73493b6dd6697b1dec65f4ae9ad0744a969e4e54362a2837a56ead3d77"}
Sep 16 18:50:50.494088 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:50.494054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hk4b6" event={"ID":"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa","Type":"ContainerStarted","Data":"ef9c498992851199e70cc00d9138c70b56cce080a7646545443ea604ef8ed093"}
Sep 16 18:50:50.496080 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:50.496055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t" event={"ID":"4bbe122e-7269-44c8-8c65-215c5dcac3ad","Type":"ContainerStarted","Data":"565102f527339b706e84068c924ac191140ac5cf1ce8353fb84c8f76652dadc7"}
Sep 16 18:50:54.512529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.512493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-hk4b6" event={"ID":"3ca16888-6858-4eb3-b9ff-63f23ec6f8fa","Type":"ContainerStarted","Data":"810870ef1516e8b89ce13a3aa5f7817345c13acc98489eca36ee943c46781f60"}
Sep 16 18:50:54.513010 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.512676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:50:54.514283 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.514256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t" event={"ID":"4bbe122e-7269-44c8-8c65-215c5dcac3ad","Type":"ContainerStarted","Data":"f48f4468d82044948d73dcbd6bf6a76c76712146a6fd603e34186f48d61f7bd6"}
Sep 16 18:50:54.514404 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.514346    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:50:54.515733 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.515713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt" event={"ID":"7d1be23c-0efb-474a-af87-6506d4f54984","Type":"ContainerStarted","Data":"4c2dc2407b3e51d228f00f026a4d840679bdf40802fef800cf3607d401fd0f7a"}
Sep 16 18:50:54.515929 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.515899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:50:54.531513 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.531463    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-hk4b6" podStartSLOduration=3.883388193 podStartE2EDuration="7.531448678s" totalImagesPullingTime="3.648060485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:54.52940239 +0000 UTC m=+355.833419381" watchObservedRunningTime="2026-09-16 18:50:54.531448678 +0000 UTC m=+355.835465644"
Sep 16 18:50:54.546962 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.546908    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t" podStartSLOduration=2.636151763 podStartE2EDuration="6.546889618s" totalImagesPullingTime="3.910737855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:54.545300815 +0000 UTC m=+355.849317781" watchObservedRunningTime="2026-09-16 18:50:54.546889618 +0000 UTC m=+355.850906585"
Sep 16 18:50:54.562094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:50:54.562043    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt" podStartSLOduration=3.145712303 podStartE2EDuration="6.562027798s" totalImagesPullingTime="3.416315495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:54.560836671 +0000 UTC m=+355.864853657" watchObservedRunningTime="2026-09-16 18:50:54.562027798 +0000 UTC m=+355.866044779"
Sep 16 18:51:05.525385 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:05.525224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-nlm6t"
Sep 16 18:51:15.521427 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:15.521396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-hk4b6"
Sep 16 18:51:15.521931 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:15.521475    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-q6qrt"
Sep 16 18:51:40.774949 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.774917    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"]
Sep 16 18:51:40.780237 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.780215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:40.783138 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.783121    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:51:40.783208 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.783127    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:51:40.784176 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.784163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vp8rv\""
Sep 16 18:51:40.788058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.788037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"]
Sep 16 18:51:40.969025 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.968994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:40.969179 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.969030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/2c359448-e77c-4fce-a3eb-55c87586c981-kube-api-access-bc225\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:40.969179 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:40.969127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.070303 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.070235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.070303 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.070270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/2c359448-e77c-4fce-a3eb-55c87586c981-kube-api-access-bc225\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.070523 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.070311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.070603 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.070586    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.070638 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.070626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.080930 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.080900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/2c359448-e77c-4fce-a3eb-55c87586c981-kube-api-access-bc225\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.089689 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.089671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:41.357188 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.357155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"]
Sep 16 18:51:41.359233 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:51:41.359207    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c359448_e77c_4fce_a3eb_55c87586c981.slice/crio-6b49c12a602ee95464330799512b49e5dee56b535e89e546c0a91a513cc8d748 WatchSource:0}: Error finding container 6b49c12a602ee95464330799512b49e5dee56b535e89e546c0a91a513cc8d748: Status 404 returned error can't find the container with id 6b49c12a602ee95464330799512b49e5dee56b535e89e546c0a91a513cc8d748
Sep 16 18:51:41.686750 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.686718    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c359448-e77c-4fce-a3eb-55c87586c981" containerID="a721993ee7892cfbf2195933b0917567181373d3cb75c31907566b5d7b35bed1" exitCode=0
Sep 16 18:51:41.686908 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.686772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278" event={"ID":"2c359448-e77c-4fce-a3eb-55c87586c981","Type":"ContainerDied","Data":"a721993ee7892cfbf2195933b0917567181373d3cb75c31907566b5d7b35bed1"}
Sep 16 18:51:41.686908 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:41.686806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278" event={"ID":"2c359448-e77c-4fce-a3eb-55c87586c981","Type":"ContainerStarted","Data":"6b49c12a602ee95464330799512b49e5dee56b535e89e546c0a91a513cc8d748"}
Sep 16 18:51:44.701305 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:44.701270    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c359448-e77c-4fce-a3eb-55c87586c981" containerID="6abe8344e0e7e94d0bf7cee21741ff894a5150559253dc1459fb2dae2657839b" exitCode=0
Sep 16 18:51:44.701850 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:44.701346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278" event={"ID":"2c359448-e77c-4fce-a3eb-55c87586c981","Type":"ContainerDied","Data":"6abe8344e0e7e94d0bf7cee21741ff894a5150559253dc1459fb2dae2657839b"}
Sep 16 18:51:45.706657 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:45.706621    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c359448-e77c-4fce-a3eb-55c87586c981" containerID="6aef5a1df89f162a32ef08f7c225aab6e83c3a5d06f27128c1ce12a7f9ee5bf5" exitCode=0
Sep 16 18:51:45.707047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:45.706705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278" event={"ID":"2c359448-e77c-4fce-a3eb-55c87586c981","Type":"ContainerDied","Data":"6aef5a1df89f162a32ef08f7c225aab6e83c3a5d06f27128c1ce12a7f9ee5bf5"}
Sep 16 18:51:46.826179 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.826158    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:46.910693 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.910664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-util\") pod \"2c359448-e77c-4fce-a3eb-55c87586c981\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") "
Sep 16 18:51:46.910882 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.910713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-bundle\") pod \"2c359448-e77c-4fce-a3eb-55c87586c981\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") "
Sep 16 18:51:46.910882 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.910740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/2c359448-e77c-4fce-a3eb-55c87586c981-kube-api-access-bc225\") pod \"2c359448-e77c-4fce-a3eb-55c87586c981\" (UID: \"2c359448-e77c-4fce-a3eb-55c87586c981\") "
Sep 16 18:51:46.911358 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.911330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-bundle" (OuterVolumeSpecName: "bundle") pod "2c359448-e77c-4fce-a3eb-55c87586c981" (UID: "2c359448-e77c-4fce-a3eb-55c87586c981"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:46.912716 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.912694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c359448-e77c-4fce-a3eb-55c87586c981-kube-api-access-bc225" (OuterVolumeSpecName: "kube-api-access-bc225") pod "2c359448-e77c-4fce-a3eb-55c87586c981" (UID: "2c359448-e77c-4fce-a3eb-55c87586c981"). InnerVolumeSpecName "kube-api-access-bc225". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:46.915204 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:46.915176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-util" (OuterVolumeSpecName: "util") pod "2c359448-e77c-4fce-a3eb-55c87586c981" (UID: "2c359448-e77c-4fce-a3eb-55c87586c981"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:47.012306 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:47.012223    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:51:47.012306 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:47.012260    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c359448-e77c-4fce-a3eb-55c87586c981-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:51:47.012306 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:47.012272    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/2c359448-e77c-4fce-a3eb-55c87586c981-kube-api-access-bc225\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:51:47.715012 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:47.714975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278" event={"ID":"2c359448-e77c-4fce-a3eb-55c87586c981","Type":"ContainerDied","Data":"6b49c12a602ee95464330799512b49e5dee56b535e89e546c0a91a513cc8d748"}
Sep 16 18:51:47.715168 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:47.715021    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b49c12a602ee95464330799512b49e5dee56b535e89e546c0a91a513cc8d748"
Sep 16 18:51:47.715168 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:47.714988    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675408d278"
Sep 16 18:51:53.661907 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.661875    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"]
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662102    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="util"
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662111    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="util"
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662129    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="pull"
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662134    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="pull"
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662141    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="extract"
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662148    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="extract"
Sep 16 18:51:53.662259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.662186    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c359448-e77c-4fce-a3eb-55c87586c981" containerName="extract"
Sep 16 18:51:53.664960 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.664939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.671305 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.671277    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:51:53.671416 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.671384    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 18:51:53.671943 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.671918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vkj24\""
Sep 16 18:51:53.682000 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.681980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"]
Sep 16 18:51:53.753295 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.753267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be425827-b745-4bf5-b00c-cc53dd88a814-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h48lg\" (UID: \"be425827-b745-4bf5-b00c-cc53dd88a814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.753374 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.753309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lwt\" (UniqueName: \"kubernetes.io/projected/be425827-b745-4bf5-b00c-cc53dd88a814-kube-api-access-x6lwt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h48lg\" (UID: \"be425827-b745-4bf5-b00c-cc53dd88a814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.854575 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.854544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be425827-b745-4bf5-b00c-cc53dd88a814-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h48lg\" (UID: \"be425827-b745-4bf5-b00c-cc53dd88a814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.854674 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.854591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lwt\" (UniqueName: \"kubernetes.io/projected/be425827-b745-4bf5-b00c-cc53dd88a814-kube-api-access-x6lwt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h48lg\" (UID: \"be425827-b745-4bf5-b00c-cc53dd88a814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.855003 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.854985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be425827-b745-4bf5-b00c-cc53dd88a814-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h48lg\" (UID: \"be425827-b745-4bf5-b00c-cc53dd88a814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.865578 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.865558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lwt\" (UniqueName: \"kubernetes.io/projected/be425827-b745-4bf5-b00c-cc53dd88a814-kube-api-access-x6lwt\") pod \"cert-manager-operator-controller-manager-684f79c9b9-h48lg\" (UID: \"be425827-b745-4bf5-b00c-cc53dd88a814\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:53.973755 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:53.973694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"
Sep 16 18:51:54.252415 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:54.252312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg"]
Sep 16 18:51:54.255605 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:51:54.255575    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe425827_b745_4bf5_b00c_cc53dd88a814.slice/crio-db175cf8ad89a501534d80a033c7170e39fad0ff73c0910af2efb3835473d48e WatchSource:0}: Error finding container db175cf8ad89a501534d80a033c7170e39fad0ff73c0910af2efb3835473d48e: Status 404 returned error can't find the container with id db175cf8ad89a501534d80a033c7170e39fad0ff73c0910af2efb3835473d48e
Sep 16 18:51:54.742072 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:54.742038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg" event={"ID":"be425827-b745-4bf5-b00c-cc53dd88a814","Type":"ContainerStarted","Data":"db175cf8ad89a501534d80a033c7170e39fad0ff73c0910af2efb3835473d48e"}
Sep 16 18:51:56.753369 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:56.753333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg" event={"ID":"be425827-b745-4bf5-b00c-cc53dd88a814","Type":"ContainerStarted","Data":"43c8e0bf2b2355ae5357be1869172c6d31e272564572536b842a14548e27f66b"}
Sep 16 18:51:56.774943 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:51:56.774894    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-h48lg" podStartSLOduration=2.191758293 podStartE2EDuration="3.774875325s" totalImagesPullingTime="1.583117032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:56.772710368 +0000 UTC m=+418.076727335" watchObservedRunningTime="2026-09-16 18:51:56.774875325 +0000 UTC m=+418.078892293"
Sep 16 18:52:02.893520 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.893490    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"]
Sep 16 18:52:02.896615 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.896596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:02.899227 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.899206    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:52:02.899227 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.899224    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vp8rv\""
Sep 16 18:52:02.900350 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.900332    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:52:02.904793 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.904774    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"]
Sep 16 18:52:02.925207 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.925178    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nwl5\" (UniqueName: \"kubernetes.io/projected/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-kube-api-access-2nwl5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:02.925340 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.925218    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:02.925340 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:02.925279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.026303 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.026266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.026469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.026329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.026469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.026390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nwl5\" (UniqueName: \"kubernetes.io/projected/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-kube-api-access-2nwl5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.026748 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.026727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.026783 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.026747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.035016 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.034981    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nwl5\" (UniqueName: \"kubernetes.io/projected/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-kube-api-access-2nwl5\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.205456 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.205429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:03.477928 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.477843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"]
Sep 16 18:52:03.480571 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:52:03.480545    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17819d31_1d62_4a2b_8e7e_f2cf2f09a2c3.slice/crio-0e57f562124c95a3fea88e428d7a631261a226adf9eebd8e5791ec7e8f8d8042 WatchSource:0}: Error finding container 0e57f562124c95a3fea88e428d7a631261a226adf9eebd8e5791ec7e8f8d8042: Status 404 returned error can't find the container with id 0e57f562124c95a3fea88e428d7a631261a226adf9eebd8e5791ec7e8f8d8042
Sep 16 18:52:03.786518 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.786440    2581 generic.go:356] "Generic (PLEG): container finished" podID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerID="98f5b557500a60378b529d2e42d1de3643b0d66fb8ee6235b0820bc6ca6693ee" exitCode=0
Sep 16 18:52:03.786658 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.786522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr" event={"ID":"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3","Type":"ContainerDied","Data":"98f5b557500a60378b529d2e42d1de3643b0d66fb8ee6235b0820bc6ca6693ee"}
Sep 16 18:52:03.786658 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:03.786598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr" event={"ID":"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3","Type":"ContainerStarted","Data":"0e57f562124c95a3fea88e428d7a631261a226adf9eebd8e5791ec7e8f8d8042"}
Sep 16 18:52:06.799928 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:06.799887    2581 generic.go:356] "Generic (PLEG): container finished" podID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerID="29c370c570fabed1c0f633a434a57bf1806aa09c020321d00ac968028c556a0c" exitCode=0
Sep 16 18:52:06.800291 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:06.799978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr" event={"ID":"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3","Type":"ContainerDied","Data":"29c370c570fabed1c0f633a434a57bf1806aa09c020321d00ac968028c556a0c"}
Sep 16 18:52:07.807141 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:07.807109    2581 generic.go:356] "Generic (PLEG): container finished" podID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerID="76e36db278f31c2ce40d042c0e73720faa8a96610290cba0a9656c7b353f3db9" exitCode=0
Sep 16 18:52:07.807468 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:07.807163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr" event={"ID":"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3","Type":"ContainerDied","Data":"76e36db278f31c2ce40d042c0e73720faa8a96610290cba0a9656c7b353f3db9"}
Sep 16 18:52:08.932989 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.932968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:08.967038 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.967016    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-bundle\") pod \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") "
Sep 16 18:52:08.967166 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.967071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nwl5\" (UniqueName: \"kubernetes.io/projected/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-kube-api-access-2nwl5\") pod \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") "
Sep 16 18:52:08.967166 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.967109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-util\") pod \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\" (UID: \"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3\") "
Sep 16 18:52:08.967421 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.967399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-bundle" (OuterVolumeSpecName: "bundle") pod "17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" (UID: "17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:08.969097 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.969068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-kube-api-access-2nwl5" (OuterVolumeSpecName: "kube-api-access-2nwl5") pod "17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" (UID: "17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3"). InnerVolumeSpecName "kube-api-access-2nwl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:08.971505 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:08.971464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-util" (OuterVolumeSpecName: "util") pod "17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" (UID: "17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:52:09.068039 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:09.067956    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nwl5\" (UniqueName: \"kubernetes.io/projected/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-kube-api-access-2nwl5\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:52:09.068039 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:09.067985    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:52:09.068039 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:09.067996    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:52:09.818359 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:09.818320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr" event={"ID":"17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3","Type":"ContainerDied","Data":"0e57f562124c95a3fea88e428d7a631261a226adf9eebd8e5791ec7e8f8d8042"}
Sep 16 18:52:09.818359 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:09.818343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfzqr"
Sep 16 18:52:09.818559 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:09.818403    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e57f562124c95a3fea88e428d7a631261a226adf9eebd8e5791ec7e8f8d8042"
Sep 16 18:52:41.033146 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033110    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"]
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033411    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="pull"
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033424    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="pull"
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033438    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="util"
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033443    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="util"
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033449    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="extract"
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033455    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="extract"
Sep 16 18:52:41.033577 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.033498    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="17819d31-1d62-4a2b-8e7e-f2cf2f09a2c3" containerName="extract"
Sep 16 18:52:41.036266 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.036248    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.040389 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.040367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:52:41.040526 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.040472    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:52:41.040607 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.040589    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:52:41.040738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.040721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-df277\""
Sep 16 18:52:41.041471 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.041457    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:52:41.077415 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.077391    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"]
Sep 16 18:52:41.095098 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.095194 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.095194 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.095194 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.095194 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.095323 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbdg\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-kube-api-access-9mbdg\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.095323 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.095298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196520 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbdg\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-kube-api-access-9mbdg\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196529    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.196653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.196639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.197255 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.197223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.199050 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.199021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.199151 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.199066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.199151 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.199135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.199261 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.199150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.205294 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.205271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbdg\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-kube-api-access-9mbdg\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.205423 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.205370    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ld2zb\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.345045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.344963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:41.625601 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.625577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"]
Sep 16 18:52:41.627651 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:52:41.627618    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5cd7cf_9dfc_40c5_9f7d_57ae4e622368.slice/crio-bd2698105b8b7fe60f3387d2bdbb4d3355864bf58ba224129e1a5cb65b552f4b WatchSource:0}: Error finding container bd2698105b8b7fe60f3387d2bdbb4d3355864bf58ba224129e1a5cb65b552f4b: Status 404 returned error can't find the container with id bd2698105b8b7fe60f3387d2bdbb4d3355864bf58ba224129e1a5cb65b552f4b
Sep 16 18:52:41.942239 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:41.942204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" event={"ID":"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368","Type":"ContainerStarted","Data":"bd2698105b8b7fe60f3387d2bdbb4d3355864bf58ba224129e1a5cb65b552f4b"}
Sep 16 18:52:48.751006 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:48.750954    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:48.751236 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:48.751044    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:48.973581 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:48.973533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" event={"ID":"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368","Type":"ContainerStarted","Data":"6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041"}
Sep 16 18:52:48.973740 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:48.973680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:48.995016 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:48.994973    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" podStartSLOduration=1.8738264770000002 podStartE2EDuration="8.994958213s" totalImagesPullingTime="7.121131736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:48.993750685 +0000 UTC m=+470.297767665" watchObservedRunningTime="2026-09-16 18:52:48.994958213 +0000 UTC m=+470.298975179"
Sep 16 18:52:49.978926 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:49.978889    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-ld2zb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:52:49.979284 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:49.978941    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" podUID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:52:52.978371 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:52.978342    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:52:53.460103 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.460073    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"]
Sep 16 18:52:53.463155 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.463132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.465936 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.465900    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-phdbh\""
Sep 16 18:52:53.477883 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.477844    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"]
Sep 16 18:52:53.587767 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.587731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.587986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.587774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.587986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.587792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.587986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.587809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.587986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.587925    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.587986 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.587965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.588189 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.588021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6de2c0-b034-4e27-8331-8113e71873a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.588189 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.588054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rst6r\" (UniqueName: \"kubernetes.io/projected/9c6de2c0-b034-4e27-8331-8113e71873a5-kube-api-access-rst6r\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.588189 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.588072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689622 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689622 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689622 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689622 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689622 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6de2c0-b034-4e27-8331-8113e71873a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rst6r\" (UniqueName: \"kubernetes.io/projected/9c6de2c0-b034-4e27-8331-8113e71873a5-kube-api-access-rst6r\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.689902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.690073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.689903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.690073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.690002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.690140 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.690066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.690487 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.690467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6de2c0-b034-4e27-8331-8113e71873a5-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.691832 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.691811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.692161 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.692143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.697394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.697375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6de2c0-b034-4e27-8331-8113e71873a5-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.697547 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.697529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rst6r\" (UniqueName: \"kubernetes.io/projected/9c6de2c0-b034-4e27-8331-8113e71873a5-kube-api-access-rst6r\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-5p4nm\" (UID: \"9c6de2c0-b034-4e27-8331-8113e71873a5\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:53.775229 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:53.775152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:54.048355 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:54.048329    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"]
Sep 16 18:52:54.050401 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:52:54.050373    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6de2c0_b034_4e27_8331_8113e71873a5.slice/crio-8bcec35878614db6a501f3672f92192aab3d3ffe5620090d8863803531dfd25c WatchSource:0}: Error finding container 8bcec35878614db6a501f3672f92192aab3d3ffe5620090d8863803531dfd25c: Status 404 returned error can't find the container with id 8bcec35878614db6a501f3672f92192aab3d3ffe5620090d8863803531dfd25c
Sep 16 18:52:54.998220 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:54.998179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" event={"ID":"9c6de2c0-b034-4e27-8331-8113e71873a5","Type":"ContainerStarted","Data":"8bcec35878614db6a501f3672f92192aab3d3ffe5620090d8863803531dfd25c"}
Sep 16 18:52:56.579528 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:56.579491    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:56.579806 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:56.579557    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:56.579806 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:56.579592    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:52:57.009683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:57.009647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" event={"ID":"9c6de2c0-b034-4e27-8331-8113e71873a5","Type":"ContainerStarted","Data":"d7dfcd134816e676cb3e11312f0f8996019b70f5720dc50289f4e9d3421225df"}
Sep 16 18:52:57.031200 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:57.031146    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podStartSLOduration=1.5046646479999999 podStartE2EDuration="4.03113095s" totalImagesPullingTime="2.526466302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:57.029510626 +0000 UTC m=+478.333527590" watchObservedRunningTime="2026-09-16 18:52:57.03113095 +0000 UTC m=+478.335147916"
Sep 16 18:52:57.775872 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:57.775816    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:57.780987 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:57.780960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:58.013585 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:58.013551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:52:58.014509 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:52:58.014485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:53:02.209641 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.209605    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"]
Sep 16 18:53:02.213032 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.213014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.215609 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.215584    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:53:02.217468 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.217451    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:53:02.217605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.217457    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vp8rv\""
Sep 16 18:53:02.221512 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.221490    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"]
Sep 16 18:53:02.309529 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.309498    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"]
Sep 16 18:53:02.312768 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.312752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.321276 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.321245    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"]
Sep 16 18:53:02.365243 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.365206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6hfn\" (UniqueName: \"kubernetes.io/projected/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-kube-api-access-z6hfn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.365408 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.365257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.365408 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.365323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.414231 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.414200    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"]
Sep 16 18:53:02.417669 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.417652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.428828 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.428788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"]
Sep 16 18:53:02.466628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.466536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.466628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.466581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv9zl\" (UniqueName: \"kubernetes.io/projected/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-kube-api-access-fv9zl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.466628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.466622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.466839 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.466696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6hfn\" (UniqueName: \"kubernetes.io/projected/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-kube-api-access-z6hfn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.466839 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.466732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.466839 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.466775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.467243 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.467224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.467279 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.467245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.475279 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.475254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6hfn\" (UniqueName: \"kubernetes.io/projected/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-kube-api-access-z6hfn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.514973 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.514940    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"]
Sep 16 18:53:02.518403 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.518387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.522360 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.522345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:02.527475 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.527451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"]
Sep 16 18:53:02.568376 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.568258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.568376 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.568323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.568376 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.568350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv9zl\" (UniqueName: \"kubernetes.io/projected/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-kube-api-access-fv9zl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.568685 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.568396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.568825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.568798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.569332 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.569091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.569596 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.568815    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qrt\" (UniqueName: \"kubernetes.io/projected/09a75dea-cc85-4033-b00b-c4a99b7e097c-kube-api-access-l7qrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.570014 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.569989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.577636 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.577604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv9zl\" (UniqueName: \"kubernetes.io/projected/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-kube-api-access-fv9zl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.622355 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.622323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:02.671182 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qrt\" (UniqueName: \"kubernetes.io/projected/09a75dea-cc85-4033-b00b-c4a99b7e097c-kube-api-access-l7qrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.671285 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.671399 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.671467 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.671570 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.671632 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snz58\" (UniqueName: \"kubernetes.io/projected/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-kube-api-access-snz58\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.671683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.671980 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.671924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.681842 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.681816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qrt\" (UniqueName: \"kubernetes.io/projected/09a75dea-cc85-4033-b00b-c4a99b7e097c-kube-api-access-l7qrt\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.727876 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.727768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:02.772198 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.772165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snz58\" (UniqueName: \"kubernetes.io/projected/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-kube-api-access-snz58\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.772378 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.772218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.772378 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.772249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.772660 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.772626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.772748 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.772684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.784130 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.783890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snz58\" (UniqueName: \"kubernetes.io/projected/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-kube-api-access-snz58\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.800005 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.799975    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"]
Sep 16 18:53:02.801495 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:53:02.801469    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f98d7d_eed9_4619_85f7_bb3d5c082f66.slice/crio-54dd04ad4c75980934a5a162c863531ec2ddd073b10af4fcb4208cc0e7345e2d WatchSource:0}: Error finding container 54dd04ad4c75980934a5a162c863531ec2ddd073b10af4fcb4208cc0e7345e2d: Status 404 returned error can't find the container with id 54dd04ad4c75980934a5a162c863531ec2ddd073b10af4fcb4208cc0e7345e2d
Sep 16 18:53:02.828352 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.828322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:02.894528 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:02.894500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"]
Sep 16 18:53:02.896729 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:53:02.896692    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b77bad5_6aaf_4f40_b5d9_c22b32f9f221.slice/crio-ee4788d1dade30eda7840a855761493233b38d0ca42293d1cf0139c53aa4ed78 WatchSource:0}: Error finding container ee4788d1dade30eda7840a855761493233b38d0ca42293d1cf0139c53aa4ed78: Status 404 returned error can't find the container with id ee4788d1dade30eda7840a855761493233b38d0ca42293d1cf0139c53aa4ed78
Sep 16 18:53:03.000383 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.000272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"]
Sep 16 18:53:03.002570 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:53:03.002543    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a75dea_cc85_4033_b00b_c4a99b7e097c.slice/crio-0177d4ae46a86ef7d1b34786f3008af66dbdb6909c0a92b158f87ce7c130e445 WatchSource:0}: Error finding container 0177d4ae46a86ef7d1b34786f3008af66dbdb6909c0a92b158f87ce7c130e445: Status 404 returned error can't find the container with id 0177d4ae46a86ef7d1b34786f3008af66dbdb6909c0a92b158f87ce7c130e445
Sep 16 18:53:03.037437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.037407    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerID="71d9b89cab4d674e1938c68a38e3f6ed3aa99c20140f9653affafb336eb7d7b6" exitCode=0
Sep 16 18:53:03.037563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.037478    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z" event={"ID":"f0f98d7d-eed9-4619-85f7-bb3d5c082f66","Type":"ContainerDied","Data":"71d9b89cab4d674e1938c68a38e3f6ed3aa99c20140f9653affafb336eb7d7b6"}
Sep 16 18:53:03.037563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.037530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z" event={"ID":"f0f98d7d-eed9-4619-85f7-bb3d5c082f66","Type":"ContainerStarted","Data":"54dd04ad4c75980934a5a162c863531ec2ddd073b10af4fcb4208cc0e7345e2d"}
Sep 16 18:53:03.038906 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.038875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b" event={"ID":"09a75dea-cc85-4033-b00b-c4a99b7e097c","Type":"ContainerStarted","Data":"0177d4ae46a86ef7d1b34786f3008af66dbdb6909c0a92b158f87ce7c130e445"}
Sep 16 18:53:03.040403 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.040380    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerID="eb4e3a1d4d93b8a231476023234fbc6c325c71c0490fee76bf5d61ed578411b0" exitCode=0
Sep 16 18:53:03.040485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.040410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2" event={"ID":"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221","Type":"ContainerDied","Data":"eb4e3a1d4d93b8a231476023234fbc6c325c71c0490fee76bf5d61ed578411b0"}
Sep 16 18:53:03.040485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.040449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2" event={"ID":"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221","Type":"ContainerStarted","Data":"ee4788d1dade30eda7840a855761493233b38d0ca42293d1cf0139c53aa4ed78"}
Sep 16 18:53:03.103149 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:03.103121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"]
Sep 16 18:53:03.105605 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:53:03.105576    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a2f5f8_2e17_4447_b8bd_9f8079ca0699.slice/crio-38d2a40565a6725cadb59d1aecabdf7742afd957f71a3ea88d31f1e9d8c5fa61 WatchSource:0}: Error finding container 38d2a40565a6725cadb59d1aecabdf7742afd957f71a3ea88d31f1e9d8c5fa61: Status 404 returned error can't find the container with id 38d2a40565a6725cadb59d1aecabdf7742afd957f71a3ea88d31f1e9d8c5fa61
Sep 16 18:53:04.047418 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.047336    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerID="ab1ff7d7503f309691120cf9fc9a34b9576e5694b4277254dc99869669186ebb" exitCode=0
Sep 16 18:53:04.047823 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.047418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z" event={"ID":"f0f98d7d-eed9-4619-85f7-bb3d5c082f66","Type":"ContainerDied","Data":"ab1ff7d7503f309691120cf9fc9a34b9576e5694b4277254dc99869669186ebb"}
Sep 16 18:53:04.049058 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.049039    2581 generic.go:356] "Generic (PLEG): container finished" podID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerID="749ef7bdf482c071ed647d8f5284c2575162d9f0463641012d0a86f0450f4299" exitCode=0
Sep 16 18:53:04.049161 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.049129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b" event={"ID":"09a75dea-cc85-4033-b00b-c4a99b7e097c","Type":"ContainerDied","Data":"749ef7bdf482c071ed647d8f5284c2575162d9f0463641012d0a86f0450f4299"}
Sep 16 18:53:04.050619 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.050597    2581 generic.go:356] "Generic (PLEG): container finished" podID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerID="a86f7824c32f897d5a460f3bf0f73ebd3906cda0101696e1a666482f50ecd369" exitCode=0
Sep 16 18:53:04.050710 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.050676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt" event={"ID":"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699","Type":"ContainerDied","Data":"a86f7824c32f897d5a460f3bf0f73ebd3906cda0101696e1a666482f50ecd369"}
Sep 16 18:53:04.050769 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.050715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt" event={"ID":"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699","Type":"ContainerStarted","Data":"38d2a40565a6725cadb59d1aecabdf7742afd957f71a3ea88d31f1e9d8c5fa61"}
Sep 16 18:53:04.052796 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.052770    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerID="83908a5f8bfe75d646cebb282251f1c9d0c4d9197112161240ef7f5214b283e3" exitCode=0
Sep 16 18:53:04.052885 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:04.052846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2" event={"ID":"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221","Type":"ContainerDied","Data":"83908a5f8bfe75d646cebb282251f1c9d0c4d9197112161240ef7f5214b283e3"}
Sep 16 18:53:05.059580 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:05.059542    2581 generic.go:356] "Generic (PLEG): container finished" podID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerID="e13ef36a8c5870b58e049d67cff1621f5f9994a8b0269bed678909ea79fdbea1" exitCode=0
Sep 16 18:53:05.060037 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:05.059589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2" event={"ID":"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221","Type":"ContainerDied","Data":"e13ef36a8c5870b58e049d67cff1621f5f9994a8b0269bed678909ea79fdbea1"}
Sep 16 18:53:05.061757 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:05.061734    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerID="4da947b2aed5e457fa2e4f8c84270cfb93a3fef4e1a629319c7c961d28981b60" exitCode=0
Sep 16 18:53:05.061886 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:05.061810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z" event={"ID":"f0f98d7d-eed9-4619-85f7-bb3d5c082f66","Type":"ContainerDied","Data":"4da947b2aed5e457fa2e4f8c84270cfb93a3fef4e1a629319c7c961d28981b60"}
Sep 16 18:53:05.063511 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:05.063489    2581 generic.go:356] "Generic (PLEG): container finished" podID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerID="49cf9bd3e00f33e52e2b271ebd8e75a50c1bb79a1a85e13108b138a3531af613" exitCode=0
Sep 16 18:53:05.063625 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:05.063547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b" event={"ID":"09a75dea-cc85-4033-b00b-c4a99b7e097c","Type":"ContainerDied","Data":"49cf9bd3e00f33e52e2b271ebd8e75a50c1bb79a1a85e13108b138a3531af613"}
Sep 16 18:53:06.069363 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.069332    2581 generic.go:356] "Generic (PLEG): container finished" podID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerID="6624bd86d633d5430efe88c0ec23a21eb324da88d3700ac7bc2ac470ebcb5f62" exitCode=0
Sep 16 18:53:06.069823 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.069412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b" event={"ID":"09a75dea-cc85-4033-b00b-c4a99b7e097c","Type":"ContainerDied","Data":"6624bd86d633d5430efe88c0ec23a21eb324da88d3700ac7bc2ac470ebcb5f62"}
Sep 16 18:53:06.195715 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.195690    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:06.228503 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.228477    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:06.303113 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.303077    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv9zl\" (UniqueName: \"kubernetes.io/projected/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-kube-api-access-fv9zl\") pod \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") "
Sep 16 18:53:06.303281 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.303131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-bundle\") pod \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") "
Sep 16 18:53:06.303281 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.303174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-util\") pod \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\" (UID: \"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221\") "
Sep 16 18:53:06.303677 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.303650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-bundle" (OuterVolumeSpecName: "bundle") pod "3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" (UID: "3b77bad5-6aaf-4f40-b5d9-c22b32f9f221"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:06.305295 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.305276    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-kube-api-access-fv9zl" (OuterVolumeSpecName: "kube-api-access-fv9zl") pod "3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" (UID: "3b77bad5-6aaf-4f40-b5d9-c22b32f9f221"). InnerVolumeSpecName "kube-api-access-fv9zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:06.310027 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.309984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-util" (OuterVolumeSpecName: "util") pod "3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" (UID: "3b77bad5-6aaf-4f40-b5d9-c22b32f9f221"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:06.403748 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.403713    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-util\") pod \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") "
Sep 16 18:53:06.403949 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.403760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6hfn\" (UniqueName: \"kubernetes.io/projected/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-kube-api-access-z6hfn\") pod \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") "
Sep 16 18:53:06.403949 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.403826    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-bundle\") pod \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\" (UID: \"f0f98d7d-eed9-4619-85f7-bb3d5c082f66\") "
Sep 16 18:53:06.404067 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.403996    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv9zl\" (UniqueName: \"kubernetes.io/projected/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-kube-api-access-fv9zl\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:06.404067 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.404007    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:06.404067 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.404015    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b77bad5-6aaf-4f40-b5d9-c22b32f9f221-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:06.404321 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.404297    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-bundle" (OuterVolumeSpecName: "bundle") pod "f0f98d7d-eed9-4619-85f7-bb3d5c082f66" (UID: "f0f98d7d-eed9-4619-85f7-bb3d5c082f66"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:06.405904 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.405883    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-kube-api-access-z6hfn" (OuterVolumeSpecName: "kube-api-access-z6hfn") pod "f0f98d7d-eed9-4619-85f7-bb3d5c082f66" (UID: "f0f98d7d-eed9-4619-85f7-bb3d5c082f66"). InnerVolumeSpecName "kube-api-access-z6hfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:06.409195 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.409159    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-util" (OuterVolumeSpecName: "util") pod "f0f98d7d-eed9-4619-85f7-bb3d5c082f66" (UID: "f0f98d7d-eed9-4619-85f7-bb3d5c082f66"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:06.505027 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.504993    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:06.505027 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.505021    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6hfn\" (UniqueName: \"kubernetes.io/projected/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-kube-api-access-z6hfn\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:06.505027 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:06.505031    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f0f98d7d-eed9-4619-85f7-bb3d5c082f66-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:07.076749 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.076725    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2"
Sep 16 18:53:07.077163 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.076759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4clpxk2" event={"ID":"3b77bad5-6aaf-4f40-b5d9-c22b32f9f221","Type":"ContainerDied","Data":"ee4788d1dade30eda7840a855761493233b38d0ca42293d1cf0139c53aa4ed78"}
Sep 16 18:53:07.077163 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.076808    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4788d1dade30eda7840a855761493233b38d0ca42293d1cf0139c53aa4ed78"
Sep 16 18:53:07.078764 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.078743    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z"
Sep 16 18:53:07.078898 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.078782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dr42z" event={"ID":"f0f98d7d-eed9-4619-85f7-bb3d5c082f66","Type":"ContainerDied","Data":"54dd04ad4c75980934a5a162c863531ec2ddd073b10af4fcb4208cc0e7345e2d"}
Sep 16 18:53:07.078898 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.078819    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54dd04ad4c75980934a5a162c863531ec2ddd073b10af4fcb4208cc0e7345e2d"
Sep 16 18:53:07.212700 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.212673    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:07.309627 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.309590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-util\") pod \"09a75dea-cc85-4033-b00b-c4a99b7e097c\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") "
Sep 16 18:53:07.309795 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.309642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7qrt\" (UniqueName: \"kubernetes.io/projected/09a75dea-cc85-4033-b00b-c4a99b7e097c-kube-api-access-l7qrt\") pod \"09a75dea-cc85-4033-b00b-c4a99b7e097c\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") "
Sep 16 18:53:07.309795 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.309686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-bundle\") pod \"09a75dea-cc85-4033-b00b-c4a99b7e097c\" (UID: \"09a75dea-cc85-4033-b00b-c4a99b7e097c\") "
Sep 16 18:53:07.310239 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.310210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-bundle" (OuterVolumeSpecName: "bundle") pod "09a75dea-cc85-4033-b00b-c4a99b7e097c" (UID: "09a75dea-cc85-4033-b00b-c4a99b7e097c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:07.311770 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.311753    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a75dea-cc85-4033-b00b-c4a99b7e097c-kube-api-access-l7qrt" (OuterVolumeSpecName: "kube-api-access-l7qrt") pod "09a75dea-cc85-4033-b00b-c4a99b7e097c" (UID: "09a75dea-cc85-4033-b00b-c4a99b7e097c"). InnerVolumeSpecName "kube-api-access-l7qrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:07.316864 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.316833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-util" (OuterVolumeSpecName: "util") pod "09a75dea-cc85-4033-b00b-c4a99b7e097c" (UID: "09a75dea-cc85-4033-b00b-c4a99b7e097c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:07.410960 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.410933    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:07.411109 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.410964    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qrt\" (UniqueName: \"kubernetes.io/projected/09a75dea-cc85-4033-b00b-c4a99b7e097c-kube-api-access-l7qrt\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:07.411109 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:07.410977    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09a75dea-cc85-4033-b00b-c4a99b7e097c-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:08.084781 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:08.084749    2581 generic.go:356] "Generic (PLEG): container finished" podID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerID="b05f7186a128b326123e338a062b37fc0c212e1896e0aa300ad52411f40d923d" exitCode=0
Sep 16 18:53:08.085170 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:08.084833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt" event={"ID":"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699","Type":"ContainerDied","Data":"b05f7186a128b326123e338a062b37fc0c212e1896e0aa300ad52411f40d923d"}
Sep 16 18:53:08.086825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:08.086807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b" event={"ID":"09a75dea-cc85-4033-b00b-c4a99b7e097c","Type":"ContainerDied","Data":"0177d4ae46a86ef7d1b34786f3008af66dbdb6909c0a92b158f87ce7c130e445"}
Sep 16 18:53:08.086907 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:08.086834    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jnw7b"
Sep 16 18:53:08.086907 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:08.086841    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0177d4ae46a86ef7d1b34786f3008af66dbdb6909c0a92b158f87ce7c130e445"
Sep 16 18:53:09.099870 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:09.099821    2581 generic.go:356] "Generic (PLEG): container finished" podID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerID="95d36be24e06fc76f41e37739c22cb690101ade46b4d5aa87973fe31847e0330" exitCode=0
Sep 16 18:53:09.100248 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:09.099906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt" event={"ID":"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699","Type":"ContainerDied","Data":"95d36be24e06fc76f41e37739c22cb690101ade46b4d5aa87973fe31847e0330"}
Sep 16 18:53:10.233975 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.233947    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:10.334766 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.334733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-bundle\") pod \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") "
Sep 16 18:53:10.334988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.334786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snz58\" (UniqueName: \"kubernetes.io/projected/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-kube-api-access-snz58\") pod \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") "
Sep 16 18:53:10.334988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.334842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-util\") pod \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\" (UID: \"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699\") "
Sep 16 18:53:10.335425 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.335400    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-bundle" (OuterVolumeSpecName: "bundle") pod "a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" (UID: "a2a2f5f8-2e17-4447-b8bd-9f8079ca0699"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:10.337018 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.336992    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-kube-api-access-snz58" (OuterVolumeSpecName: "kube-api-access-snz58") pod "a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" (UID: "a2a2f5f8-2e17-4447-b8bd-9f8079ca0699"). InnerVolumeSpecName "kube-api-access-snz58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:53:10.339527 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.339490    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-util" (OuterVolumeSpecName: "util") pod "a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" (UID: "a2a2f5f8-2e17-4447-b8bd-9f8079ca0699"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:53:10.435233 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.435204    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snz58\" (UniqueName: \"kubernetes.io/projected/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-kube-api-access-snz58\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:10.435233 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.435232    2581 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-util\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:10.435233 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:10.435242    2581 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2a2f5f8-2e17-4447-b8bd-9f8079ca0699-bundle\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:53:11.110865 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:11.110831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt" event={"ID":"a2a2f5f8-2e17-4447-b8bd-9f8079ca0699","Type":"ContainerDied","Data":"38d2a40565a6725cadb59d1aecabdf7742afd957f71a3ea88d31f1e9d8c5fa61"}
Sep 16 18:53:11.111068 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:11.110845    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acrpjt"
Sep 16 18:53:11.111068 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:11.110893    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38d2a40565a6725cadb59d1aecabdf7742afd957f71a3ea88d31f1e9d8c5fa61"
Sep 16 18:53:22.443310 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443279    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"]
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443540    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443552    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443560    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443566    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443572    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443577    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443584    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443589    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443594    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443599    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443610    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443615    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443623    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443629    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443634    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443639    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443644    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443648    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443654    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443658    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="pull"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443681    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="util"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443725    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0f98d7d-eed9-4619-85f7-bb3d5c082f66" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443731    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a75dea-cc85-4033-b00b-c4a99b7e097c" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443737    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b77bad5-6aaf-4f40-b5d9-c22b32f9f221" containerName="extract"
Sep 16 18:53:22.443760 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.443741    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2a2f5f8-2e17-4447-b8bd-9f8079ca0699" containerName="extract"
Sep 16 18:53:22.451093 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.451065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:22.454903 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.454878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:53:22.455025 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.454918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bfjsb\""
Sep 16 18:53:22.455988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.455952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:53:22.459451 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.459429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"]
Sep 16 18:53:22.518094 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.518064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkqg9\" (UniqueName: \"kubernetes.io/projected/9efff3a4-13c3-4807-8237-fa554431d62e-kube-api-access-rkqg9\") pod \"authorino-operator-6fc8f864f4-xn9hl\" (UID: \"9efff3a4-13c3-4807-8237-fa554431d62e\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:22.618707 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.618674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkqg9\" (UniqueName: \"kubernetes.io/projected/9efff3a4-13c3-4807-8237-fa554431d62e-kube-api-access-rkqg9\") pod \"authorino-operator-6fc8f864f4-xn9hl\" (UID: \"9efff3a4-13c3-4807-8237-fa554431d62e\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:22.634389 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.634359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkqg9\" (UniqueName: \"kubernetes.io/projected/9efff3a4-13c3-4807-8237-fa554431d62e-kube-api-access-rkqg9\") pod \"authorino-operator-6fc8f864f4-xn9hl\" (UID: \"9efff3a4-13c3-4807-8237-fa554431d62e\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:22.761372 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:22.761288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:23.043800 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:23.043775    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"]
Sep 16 18:53:23.046474 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:53:23.046449    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efff3a4_13c3_4807_8237_fa554431d62e.slice/crio-b19028eee2fb255cd1b8bc0b7d1a4ff92453bc5df2813c59805420ebb424b407 WatchSource:0}: Error finding container b19028eee2fb255cd1b8bc0b7d1a4ff92453bc5df2813c59805420ebb424b407: Status 404 returned error can't find the container with id b19028eee2fb255cd1b8bc0b7d1a4ff92453bc5df2813c59805420ebb424b407
Sep 16 18:53:23.162541 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:23.162513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl" event={"ID":"9efff3a4-13c3-4807-8237-fa554431d62e","Type":"ContainerStarted","Data":"b19028eee2fb255cd1b8bc0b7d1a4ff92453bc5df2813c59805420ebb424b407"}
Sep 16 18:53:25.173794 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:25.173697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl" event={"ID":"9efff3a4-13c3-4807-8237-fa554431d62e","Type":"ContainerStarted","Data":"71a483cc5f2cd3b2a1dde5eb1a6742545a998cbd8fa266c4a2590143db70483a"}
Sep 16 18:53:25.174265 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:25.173810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:25.193226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:25.193177    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl" podStartSLOduration=1.33200464 podStartE2EDuration="3.193162995s" totalImagesPullingTime="1.861158355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:25.192095361 +0000 UTC m=+506.496112328" watchObservedRunningTime="2026-09-16 18:53:25.193162995 +0000 UTC m=+506.497179962"
Sep 16 18:53:28.864566 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:28.864532    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"]
Sep 16 18:53:28.867970 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:28.867951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:53:28.870801 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:28.870776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-42gfp\""
Sep 16 18:53:28.885904 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:28.885882    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"]
Sep 16 18:53:28.962788 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:28.962758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spp8\" (UniqueName: \"kubernetes.io/projected/116b141f-3753-488e-a002-211f731295c8-kube-api-access-5spp8\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-5xvkz\" (UID: \"116b141f-3753-488e-a002-211f731295c8\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:53:29.063230 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:29.063203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spp8\" (UniqueName: \"kubernetes.io/projected/116b141f-3753-488e-a002-211f731295c8-kube-api-access-5spp8\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-5xvkz\" (UID: \"116b141f-3753-488e-a002-211f731295c8\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:53:29.075706 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:29.075681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spp8\" (UniqueName: \"kubernetes.io/projected/116b141f-3753-488e-a002-211f731295c8-kube-api-access-5spp8\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-5xvkz\" (UID: \"116b141f-3753-488e-a002-211f731295c8\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:53:29.177562 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:29.177518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:53:29.472534 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:29.472507    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"]
Sep 16 18:53:29.474188 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:53:29.474155    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116b141f_3753_488e_a002_211f731295c8.slice/crio-3648e0cfb06730838d74d8084a17c571e5ca99ecf5e725ee8a83de717d4afe28 WatchSource:0}: Error finding container 3648e0cfb06730838d74d8084a17c571e5ca99ecf5e725ee8a83de717d4afe28: Status 404 returned error can't find the container with id 3648e0cfb06730838d74d8084a17c571e5ca99ecf5e725ee8a83de717d4afe28
Sep 16 18:53:30.196797 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:30.196766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz" event={"ID":"116b141f-3753-488e-a002-211f731295c8","Type":"ContainerStarted","Data":"3648e0cfb06730838d74d8084a17c571e5ca99ecf5e725ee8a83de717d4afe28"}
Sep 16 18:53:31.203449 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:31.203416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz" event={"ID":"116b141f-3753-488e-a002-211f731295c8","Type":"ContainerStarted","Data":"0c1332acd2b6aa3428d8f15753817e1564f9289fbf2ca802468682dca4428258"}
Sep 16 18:53:31.203840 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:31.203505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:53:31.225586 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:31.225542    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz" podStartSLOduration=1.86860498 podStartE2EDuration="3.225526756s" totalImagesPullingTime="1.356921776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:31.224162764 +0000 UTC m=+512.528179732" watchObservedRunningTime="2026-09-16 18:53:31.225526756 +0000 UTC m=+512.529543723"
Sep 16 18:53:36.181160 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:36.181131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-xn9hl"
Sep 16 18:53:42.209988 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:53:42.209960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-5xvkz"
Sep 16 18:54:17.190656 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.190578    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565v"]
Sep 16 18:54:17.195878 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.195811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.198735 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.198708    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r856j\""
Sep 16 18:54:17.199026 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.199010    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:54:17.203969 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.203516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565v"]
Sep 16 18:54:17.238929 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.238891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565v"]
Sep 16 18:54:17.308310 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.308276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2261aad0-c08c-4c48-8ceb-d452d5b21daf-config-file\") pod \"limitador-limitador-84994f7ffc-8565v\" (UID: \"2261aad0-c08c-4c48-8ceb-d452d5b21daf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.308484 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.308342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7nbf\" (UniqueName: \"kubernetes.io/projected/2261aad0-c08c-4c48-8ceb-d452d5b21daf-kube-api-access-m7nbf\") pod \"limitador-limitador-84994f7ffc-8565v\" (UID: \"2261aad0-c08c-4c48-8ceb-d452d5b21daf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.408925 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.408895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2261aad0-c08c-4c48-8ceb-d452d5b21daf-config-file\") pod \"limitador-limitador-84994f7ffc-8565v\" (UID: \"2261aad0-c08c-4c48-8ceb-d452d5b21daf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.409093 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.408946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7nbf\" (UniqueName: \"kubernetes.io/projected/2261aad0-c08c-4c48-8ceb-d452d5b21daf-kube-api-access-m7nbf\") pod \"limitador-limitador-84994f7ffc-8565v\" (UID: \"2261aad0-c08c-4c48-8ceb-d452d5b21daf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.409516 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.409495    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2261aad0-c08c-4c48-8ceb-d452d5b21daf-config-file\") pod \"limitador-limitador-84994f7ffc-8565v\" (UID: \"2261aad0-c08c-4c48-8ceb-d452d5b21daf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.416720 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.416698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7nbf\" (UniqueName: \"kubernetes.io/projected/2261aad0-c08c-4c48-8ceb-d452d5b21daf-kube-api-access-m7nbf\") pod \"limitador-limitador-84994f7ffc-8565v\" (UID: \"2261aad0-c08c-4c48-8ceb-d452d5b21daf\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.508587 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.508494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:17.777974 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:17.777945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-8565v"]
Sep 16 18:54:17.780336 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:54:17.780308    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2261aad0_c08c_4c48_8ceb_d452d5b21daf.slice/crio-7de185d2a08a1f5c67c2fc77bfb18bc6509a5d44ac5d7e08def3b9479be05814 WatchSource:0}: Error finding container 7de185d2a08a1f5c67c2fc77bfb18bc6509a5d44ac5d7e08def3b9479be05814: Status 404 returned error can't find the container with id 7de185d2a08a1f5c67c2fc77bfb18bc6509a5d44ac5d7e08def3b9479be05814
Sep 16 18:54:18.414649 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:18.414611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v" event={"ID":"2261aad0-c08c-4c48-8ceb-d452d5b21daf","Type":"ContainerStarted","Data":"7de185d2a08a1f5c67c2fc77bfb18bc6509a5d44ac5d7e08def3b9479be05814"}
Sep 16 18:54:19.420840 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:19.420805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v" event={"ID":"2261aad0-c08c-4c48-8ceb-d452d5b21daf","Type":"ContainerStarted","Data":"02f9a469db8d52d5e4a71b4dfdf37d9aafd9fb4725dc0b833e64eee6c0c88321"}
Sep 16 18:54:19.421258 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:19.420912    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:19.439023 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:19.438969    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v" podStartSLOduration=1.553440418 podStartE2EDuration="2.438952751s" totalImagesPullingTime="885.512333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:19.436064332 +0000 UTC m=+560.740081301" watchObservedRunningTime="2026-09-16 18:54:19.438952751 +0000 UTC m=+560.742969714"
Sep 16 18:54:30.427074 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:30.427043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-8565v"
Sep 16 18:54:59.140628 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:59.140602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:54:59.143780 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:59.143756    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:54:59.144287 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:59.144268    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:54:59.147425 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:54:59.147408    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:55:03.232047 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.232017    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"]
Sep 16 18:55:03.232548 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.232213    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" podUID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" containerName="discovery" containerID="cri-o://6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041" gracePeriod=30
Sep 16 18:55:03.472804 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.472775    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:55:03.575683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-dns-cert\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.575683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-kubeconfig\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.575683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575665    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mbdg\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-kube-api-access-9mbdg\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.576006 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575688    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-ca-configmap\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.576006 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-local-certs\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.576006 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-cacerts\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.576006 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.575905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-token\") pod \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\" (UID: \"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368\") "
Sep 16 18:55:03.576208 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.576178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:55:03.578192 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.578161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-local-certs" (OuterVolumeSpecName: "local-certs") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:03.578317 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.578161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:03.578377 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.578359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-kube-api-access-9mbdg" (OuterVolumeSpecName: "kube-api-access-9mbdg") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "kube-api-access-9mbdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:03.578433 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.578376    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:03.578489 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.578450    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-cacerts" (OuterVolumeSpecName: "cacerts") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:03.578669 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.578649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" (UID: "4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:03.627333 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.627306    2581 generic.go:356] "Generic (PLEG): container finished" podID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" containerID="6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041" exitCode=0
Sep 16 18:55:03.627463 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.627351    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"
Sep 16 18:55:03.627463 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.627369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" event={"ID":"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368","Type":"ContainerDied","Data":"6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041"}
Sep 16 18:55:03.627463 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.627406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb" event={"ID":"4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368","Type":"ContainerDied","Data":"bd2698105b8b7fe60f3387d2bdbb4d3355864bf58ba224129e1a5cb65b552f4b"}
Sep 16 18:55:03.627463 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.627425    2581 scope.go:117] "RemoveContainer" containerID="6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041"
Sep 16 18:55:03.636127 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.636109    2581 scope.go:117] "RemoveContainer" containerID="6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041"
Sep 16 18:55:03.636395 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:55:03.636378    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041\": container with ID starting with 6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041 not found: ID does not exist" containerID="6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041"
Sep 16 18:55:03.636437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.636406    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041"} err="failed to get container status \"6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041\": rpc error: code = NotFound desc = could not find container \"6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041\": container with ID starting with 6db86b007cacf774c545c6f04c002a2c05b33d9946d61d91a09eddadcdeb7041 not found: ID does not exist"
Sep 16 18:55:03.663498 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.663466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"]
Sep 16 18:55:03.676603 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676581    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-ca-configmap\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.676603 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676602    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-local-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.676811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676615    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-cacerts\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.676811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676627    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-token\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.676811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676637    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-csr-dns-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.676811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676659    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-istio-kubeconfig\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.676811 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.676671    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbdg\" (UniqueName: \"kubernetes.io/projected/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368-kube-api-access-9mbdg\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:03.680003 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:03.679979    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ld2zb"]
Sep 16 18:55:05.220996 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:05.220956    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" path="/var/lib/kubelet/pods/4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368/volumes"
Sep 16 18:55:11.188624 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.188592    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wfc9k"]
Sep 16 18:55:11.189055 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.188946    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" containerName="discovery"
Sep 16 18:55:11.189055 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.188961    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" containerName="discovery"
Sep 16 18:55:11.189055 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.189007    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5cd7cf-9dfc-40c5-9f7d-57ae4e622368" containerName="discovery"
Sep 16 18:55:11.193177 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.193161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.196072 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.196048    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:55:11.196185 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.196073    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:11.197144 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.197127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-mcfgc\""
Sep 16 18:55:11.197232 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.197130    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:11.200262 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.200242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wfc9k"]
Sep 16 18:55:11.210993 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.210966    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-b684f68f9-5p4wb"]
Sep 16 18:55:11.214050 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.214031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.217696 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.217678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qtfsc\""
Sep 16 18:55:11.217812 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.217721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:55:11.225803 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.225783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-b684f68f9-5p4wb"]
Sep 16 18:55:11.227686 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.227667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825fc7f4-b193-4acc-ac32-de6c8ab80600-cert\") pod \"kserve-controller-manager-764fc98f48-wfc9k\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") " pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.227772 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.227710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssf6c\" (UniqueName: \"kubernetes.io/projected/825fc7f4-b193-4acc-ac32-de6c8ab80600-kube-api-access-ssf6c\") pod \"kserve-controller-manager-764fc98f48-wfc9k\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") " pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.239210 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.239187    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-b7rzb"]
Sep 16 18:55:11.243799 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.243775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.246631 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.246616    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:55:11.246815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.246802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-pt2vh\""
Sep 16 18:55:11.251376 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.251353    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-b7rzb"]
Sep 16 18:55:11.328979 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.328952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssf6c\" (UniqueName: \"kubernetes.io/projected/825fc7f4-b193-4acc-ac32-de6c8ab80600-kube-api-access-ssf6c\") pod \"kserve-controller-manager-764fc98f48-wfc9k\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") " pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.329103 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.328992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee5a6432-9b17-462e-84a4-48319d5bca5a-data\") pod \"seaweedfs-64568bcd49-b7rzb\" (UID: \"ee5a6432-9b17-462e-84a4-48319d5bca5a\") " pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.329166 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.329148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jv7q\" (UniqueName: \"kubernetes.io/projected/ee5a6432-9b17-462e-84a4-48319d5bca5a-kube-api-access-2jv7q\") pod \"seaweedfs-64568bcd49-b7rzb\" (UID: \"ee5a6432-9b17-462e-84a4-48319d5bca5a\") " pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.329244 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.329226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825fc7f4-b193-4acc-ac32-de6c8ab80600-cert\") pod \"kserve-controller-manager-764fc98f48-wfc9k\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") " pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.329281 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.329263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52bvh\" (UniqueName: \"kubernetes.io/projected/19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab-kube-api-access-52bvh\") pod \"llmisvc-controller-manager-b684f68f9-5p4wb\" (UID: \"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab\") " pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.329314 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.329302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab-cert\") pod \"llmisvc-controller-manager-b684f68f9-5p4wb\" (UID: \"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab\") " pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.331537 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.331516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825fc7f4-b193-4acc-ac32-de6c8ab80600-cert\") pod \"kserve-controller-manager-764fc98f48-wfc9k\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") " pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.339944 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.339926    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssf6c\" (UniqueName: \"kubernetes.io/projected/825fc7f4-b193-4acc-ac32-de6c8ab80600-kube-api-access-ssf6c\") pod \"kserve-controller-manager-764fc98f48-wfc9k\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") " pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.429678 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.429652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52bvh\" (UniqueName: \"kubernetes.io/projected/19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab-kube-api-access-52bvh\") pod \"llmisvc-controller-manager-b684f68f9-5p4wb\" (UID: \"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab\") " pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.429779 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.429682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab-cert\") pod \"llmisvc-controller-manager-b684f68f9-5p4wb\" (UID: \"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab\") " pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.429779 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.429728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee5a6432-9b17-462e-84a4-48319d5bca5a-data\") pod \"seaweedfs-64568bcd49-b7rzb\" (UID: \"ee5a6432-9b17-462e-84a4-48319d5bca5a\") " pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.429779 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.429771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jv7q\" (UniqueName: \"kubernetes.io/projected/ee5a6432-9b17-462e-84a4-48319d5bca5a-kube-api-access-2jv7q\") pod \"seaweedfs-64568bcd49-b7rzb\" (UID: \"ee5a6432-9b17-462e-84a4-48319d5bca5a\") " pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.430149 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.430127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ee5a6432-9b17-462e-84a4-48319d5bca5a-data\") pod \"seaweedfs-64568bcd49-b7rzb\" (UID: \"ee5a6432-9b17-462e-84a4-48319d5bca5a\") " pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.431945 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.431929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab-cert\") pod \"llmisvc-controller-manager-b684f68f9-5p4wb\" (UID: \"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab\") " pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.438245 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.438221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jv7q\" (UniqueName: \"kubernetes.io/projected/ee5a6432-9b17-462e-84a4-48319d5bca5a-kube-api-access-2jv7q\") pod \"seaweedfs-64568bcd49-b7rzb\" (UID: \"ee5a6432-9b17-462e-84a4-48319d5bca5a\") " pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.441096 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.441052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52bvh\" (UniqueName: \"kubernetes.io/projected/19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab-kube-api-access-52bvh\") pod \"llmisvc-controller-manager-b684f68f9-5p4wb\" (UID: \"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab\") " pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.504077 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.504052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:11.525137 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.525114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:11.552531 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.552380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:11.775823 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.775800    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wfc9k"]
Sep 16 18:55:11.777659 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:55:11.777632    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825fc7f4_b193_4acc_ac32_de6c8ab80600.slice/crio-439990d1417d4578a2bbe0ddf6c4f32eb862f1643d3b15344e31bf9af0e300c1 WatchSource:0}: Error finding container 439990d1417d4578a2bbe0ddf6c4f32eb862f1643d3b15344e31bf9af0e300c1: Status 404 returned error can't find the container with id 439990d1417d4578a2bbe0ddf6c4f32eb862f1643d3b15344e31bf9af0e300c1
Sep 16 18:55:11.807258 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.806697    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-b684f68f9-5p4wb"]
Sep 16 18:55:11.808918 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:55:11.808892    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod19a6466e_fe4a_4fd1_9016_d6d4eb73c5ab.slice/crio-a0017b97bd7d0e85b0f6735cad9cbe085a9d7ec2dac22ba5b38d18146fa7dd3b WatchSource:0}: Error finding container a0017b97bd7d0e85b0f6735cad9cbe085a9d7ec2dac22ba5b38d18146fa7dd3b: Status 404 returned error can't find the container with id a0017b97bd7d0e85b0f6735cad9cbe085a9d7ec2dac22ba5b38d18146fa7dd3b
Sep 16 18:55:11.839353 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:11.839335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-b7rzb"]
Sep 16 18:55:11.840431 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:55:11.840410    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5a6432_9b17_462e_84a4_48319d5bca5a.slice/crio-8f9043366f1851054fb0a0006493513e0044cc4da9c8fedea4dc34412c48b370 WatchSource:0}: Error finding container 8f9043366f1851054fb0a0006493513e0044cc4da9c8fedea4dc34412c48b370: Status 404 returned error can't find the container with id 8f9043366f1851054fb0a0006493513e0044cc4da9c8fedea4dc34412c48b370
Sep 16 18:55:12.676407 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:12.676369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k" event={"ID":"825fc7f4-b193-4acc-ac32-de6c8ab80600","Type":"ContainerStarted","Data":"439990d1417d4578a2bbe0ddf6c4f32eb862f1643d3b15344e31bf9af0e300c1"}
Sep 16 18:55:12.679966 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:12.679886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-b7rzb" event={"ID":"ee5a6432-9b17-462e-84a4-48319d5bca5a","Type":"ContainerStarted","Data":"8f9043366f1851054fb0a0006493513e0044cc4da9c8fedea4dc34412c48b370"}
Sep 16 18:55:12.682223 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:12.682157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb" event={"ID":"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab","Type":"ContainerStarted","Data":"a0017b97bd7d0e85b0f6735cad9cbe085a9d7ec2dac22ba5b38d18146fa7dd3b"}
Sep 16 18:55:16.709364 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.709326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k" event={"ID":"825fc7f4-b193-4acc-ac32-de6c8ab80600","Type":"ContainerStarted","Data":"2aa9aebf5ef858fffe86fa494f89ef6e3ef0827e5f79121f4010ab12d9821831"}
Sep 16 18:55:16.709828 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.709449    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:16.710844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.710823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-b7rzb" event={"ID":"ee5a6432-9b17-462e-84a4-48319d5bca5a","Type":"ContainerStarted","Data":"582044f33c49b756d31436603ee915cce24e06cc2ee764e4108144eebcbeaf33"}
Sep 16 18:55:16.710953 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.710887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:16.712375 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.712350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb" event={"ID":"19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab","Type":"ContainerStarted","Data":"49b9dcd4fabf6a80cee855fa606723676cfb81ebc56bcd7b0b8acee38df49482"}
Sep 16 18:55:16.712454 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.712391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:16.726318 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.726272    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k" podStartSLOduration=1.369492885 podStartE2EDuration="5.726258573s" totalImagesPullingTime="4.356765688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:16.724685762 +0000 UTC m=+618.028702729" watchObservedRunningTime="2026-09-16 18:55:16.726258573 +0000 UTC m=+618.030275552"
Sep 16 18:55:16.742258 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.742211    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-b7rzb" podStartSLOduration=1.213728076 podStartE2EDuration="5.742193448s" totalImagesPullingTime="4.528465372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:16.740643396 +0000 UTC m=+618.044660364" watchObservedRunningTime="2026-09-16 18:55:16.742193448 +0000 UTC m=+618.046210415"
Sep 16 18:55:16.756814 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:16.756774    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb" podStartSLOduration=1.394353965 podStartE2EDuration="5.756762871s" totalImagesPullingTime="4.362408906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:16.756191013 +0000 UTC m=+618.060207981" watchObservedRunningTime="2026-09-16 18:55:16.756762871 +0000 UTC m=+618.060779842"
Sep 16 18:55:22.719380 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:22.719301    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-b7rzb"
Sep 16 18:55:47.718637 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:47.718603    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-b684f68f9-5p4wb"
Sep 16 18:55:47.721605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:47.721582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:54.735928 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.735890    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wfc9k"]
Sep 16 18:55:54.736393 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.736129    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k" podUID="825fc7f4-b193-4acc-ac32-de6c8ab80600" containerName="manager" containerID="cri-o://2aa9aebf5ef858fffe86fa494f89ef6e3ef0827e5f79121f4010ab12d9821831" gracePeriod=10
Sep 16 18:55:54.759683 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.759657    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-d58bz"]
Sep 16 18:55:54.761587 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.761569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:54.772761 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.772738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-d58bz"]
Sep 16 18:55:54.860271 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.860245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd94dacb-852f-4d4b-9c99-4736fa00044e-cert\") pod \"kserve-controller-manager-764fc98f48-d58bz\" (UID: \"dd94dacb-852f-4d4b-9c99-4736fa00044e\") " pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:54.860394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.860305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5kb7\" (UniqueName: \"kubernetes.io/projected/dd94dacb-852f-4d4b-9c99-4736fa00044e-kube-api-access-w5kb7\") pod \"kserve-controller-manager-764fc98f48-d58bz\" (UID: \"dd94dacb-852f-4d4b-9c99-4736fa00044e\") " pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:54.887528 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.887501    2581 generic.go:356] "Generic (PLEG): container finished" podID="825fc7f4-b193-4acc-ac32-de6c8ab80600" containerID="2aa9aebf5ef858fffe86fa494f89ef6e3ef0827e5f79121f4010ab12d9821831" exitCode=0
Sep 16 18:55:54.887659 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.887571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k" event={"ID":"825fc7f4-b193-4acc-ac32-de6c8ab80600","Type":"ContainerDied","Data":"2aa9aebf5ef858fffe86fa494f89ef6e3ef0827e5f79121f4010ab12d9821831"}
Sep 16 18:55:54.960818 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.960794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5kb7\" (UniqueName: \"kubernetes.io/projected/dd94dacb-852f-4d4b-9c99-4736fa00044e-kube-api-access-w5kb7\") pod \"kserve-controller-manager-764fc98f48-d58bz\" (UID: \"dd94dacb-852f-4d4b-9c99-4736fa00044e\") " pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:54.960969 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.960880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd94dacb-852f-4d4b-9c99-4736fa00044e-cert\") pod \"kserve-controller-manager-764fc98f48-d58bz\" (UID: \"dd94dacb-852f-4d4b-9c99-4736fa00044e\") " pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:54.962725 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.962702    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:54.963236 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.963215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd94dacb-852f-4d4b-9c99-4736fa00044e-cert\") pod \"kserve-controller-manager-764fc98f48-d58bz\" (UID: \"dd94dacb-852f-4d4b-9c99-4736fa00044e\") " pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:54.969328 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:54.969308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5kb7\" (UniqueName: \"kubernetes.io/projected/dd94dacb-852f-4d4b-9c99-4736fa00044e-kube-api-access-w5kb7\") pod \"kserve-controller-manager-764fc98f48-d58bz\" (UID: \"dd94dacb-852f-4d4b-9c99-4736fa00044e\") " pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:55.061452 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.061365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssf6c\" (UniqueName: \"kubernetes.io/projected/825fc7f4-b193-4acc-ac32-de6c8ab80600-kube-api-access-ssf6c\") pod \"825fc7f4-b193-4acc-ac32-de6c8ab80600\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") "
Sep 16 18:55:55.061600 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.061468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825fc7f4-b193-4acc-ac32-de6c8ab80600-cert\") pod \"825fc7f4-b193-4acc-ac32-de6c8ab80600\" (UID: \"825fc7f4-b193-4acc-ac32-de6c8ab80600\") "
Sep 16 18:55:55.063656 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.063625    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825fc7f4-b193-4acc-ac32-de6c8ab80600-cert" (OuterVolumeSpecName: "cert") pod "825fc7f4-b193-4acc-ac32-de6c8ab80600" (UID: "825fc7f4-b193-4acc-ac32-de6c8ab80600"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:55:55.063773 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.063628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825fc7f4-b193-4acc-ac32-de6c8ab80600-kube-api-access-ssf6c" (OuterVolumeSpecName: "kube-api-access-ssf6c") pod "825fc7f4-b193-4acc-ac32-de6c8ab80600" (UID: "825fc7f4-b193-4acc-ac32-de6c8ab80600"). InnerVolumeSpecName "kube-api-access-ssf6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:55.112369 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.112336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:55.162390 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.162361    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssf6c\" (UniqueName: \"kubernetes.io/projected/825fc7f4-b193-4acc-ac32-de6c8ab80600-kube-api-access-ssf6c\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.162390 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.162392    2581 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/825fc7f4-b193-4acc-ac32-de6c8ab80600-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:55:55.380039 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.380015    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-d58bz"]
Sep 16 18:55:55.381669 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:55:55.381647    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd94dacb_852f_4d4b_9c99_4736fa00044e.slice/crio-e446e9b0537bb8068a9e22d5669ae2ee4129d996304ae6660245071593b1d57a WatchSource:0}: Error finding container e446e9b0537bb8068a9e22d5669ae2ee4129d996304ae6660245071593b1d57a: Status 404 returned error can't find the container with id e446e9b0537bb8068a9e22d5669ae2ee4129d996304ae6660245071593b1d57a
Sep 16 18:55:55.382951 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.382935    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:55.900715 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.900668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-d58bz" event={"ID":"dd94dacb-852f-4d4b-9c99-4736fa00044e","Type":"ContainerStarted","Data":"4b872ecd0c231b817db887c3eba4e0e0fa02bfac1e6a3df3448770b766f988f8"}
Sep 16 18:55:55.901183 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.900759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-d58bz" event={"ID":"dd94dacb-852f-4d4b-9c99-4736fa00044e","Type":"ContainerStarted","Data":"e446e9b0537bb8068a9e22d5669ae2ee4129d996304ae6660245071593b1d57a"}
Sep 16 18:55:55.901183 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.900788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:55:55.902221 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.902197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k" event={"ID":"825fc7f4-b193-4acc-ac32-de6c8ab80600","Type":"ContainerDied","Data":"439990d1417d4578a2bbe0ddf6c4f32eb862f1643d3b15344e31bf9af0e300c1"}
Sep 16 18:55:55.902323 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.902226    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-wfc9k"
Sep 16 18:55:55.902323 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.902245    2581 scope.go:117] "RemoveContainer" containerID="2aa9aebf5ef858fffe86fa494f89ef6e3ef0827e5f79121f4010ab12d9821831"
Sep 16 18:55:55.918145 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.918100    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-764fc98f48-d58bz" podStartSLOduration=1.550299489 podStartE2EDuration="1.91808688s" totalImagesPullingTime="367.787391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:55.916049017 +0000 UTC m=+657.220065984" watchObservedRunningTime="2026-09-16 18:55:55.91808688 +0000 UTC m=+657.222103876"
Sep 16 18:55:55.931082 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.931057    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wfc9k"]
Sep 16 18:55:55.934564 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:55.934544    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wfc9k"]
Sep 16 18:55:57.221940 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:55:57.221914    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825fc7f4-b193-4acc-ac32-de6c8ab80600" path="/var/lib/kubelet/pods/825fc7f4-b193-4acc-ac32-de6c8ab80600/volumes"
Sep 16 18:56:26.912735 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:26.912704    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-764fc98f48-d58bz"
Sep 16 18:56:28.287286 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.287252    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-6kmjf"]
Sep 16 18:56:28.287644 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.287572    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825fc7f4-b193-4acc-ac32-de6c8ab80600" containerName="manager"
Sep 16 18:56:28.287644 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.287585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="825fc7f4-b193-4acc-ac32-de6c8ab80600" containerName="manager"
Sep 16 18:56:28.287713 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.287650    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="825fc7f4-b193-4acc-ac32-de6c8ab80600" containerName="manager"
Sep 16 18:56:28.290536 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.290516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.294138 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.294121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 16 18:56:28.294223 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.294171    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-dj856\""
Sep 16 18:56:28.299552 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.299533    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-6kmjf"]
Sep 16 18:56:28.303109 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.303093    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-nv2qk"]
Sep 16 18:56:28.306154 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.306139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.308789 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.308769    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:56:28.309029 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.309015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-vbkzs\""
Sep 16 18:56:28.309102 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.309083    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:56:28.317679 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.317658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nv2qk"]
Sep 16 18:56:28.415090 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.415058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jv77\" (UniqueName: \"kubernetes.io/projected/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-kube-api-access-4jv77\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.415259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.415123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-metrics-certs\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.415259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.415153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.415259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.415175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ef3166-3d20-4dd4-85d4-3b10cbc00bfd-tls-certs\") pod \"model-serving-api-7b954ffc95-6kmjf\" (UID: \"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd\") " pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.415259 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.415195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzkkc\" (UniqueName: \"kubernetes.io/projected/86ef3166-3d20-4dd4-85d4-3b10cbc00bfd-kube-api-access-zzkkc\") pod \"model-serving-api-7b954ffc95-6kmjf\" (UID: \"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd\") " pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.516340 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.516304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv77\" (UniqueName: \"kubernetes.io/projected/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-kube-api-access-4jv77\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.516521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.516375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-metrics-certs\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.516521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.516408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.516521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.516438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ef3166-3d20-4dd4-85d4-3b10cbc00bfd-tls-certs\") pod \"model-serving-api-7b954ffc95-6kmjf\" (UID: \"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd\") " pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.516521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.516460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzkkc\" (UniqueName: \"kubernetes.io/projected/86ef3166-3d20-4dd4-85d4-3b10cbc00bfd-kube-api-access-zzkkc\") pod \"model-serving-api-7b954ffc95-6kmjf\" (UID: \"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd\") " pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.516521 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:56:28.516495    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 18:56:28.516738 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:56:28.516533    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:56:28.516738 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:56:28.516575    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-metrics-certs podName:4ca058e9-7f14-41fb-b0d3-3ba5d19b0855 nodeName:}" failed. No retries permitted until 2026-09-16 18:56:29.01655847 +0000 UTC m=+690.320575415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-metrics-certs") pod "odh-model-controller-7b966d7984-nv2qk" (UID: "4ca058e9-7f14-41fb-b0d3-3ba5d19b0855") : secret "odh-model-controller-metrics-tls" not found
Sep 16 18:56:28.516738 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:56:28.516589    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert podName:4ca058e9-7f14-41fb-b0d3-3ba5d19b0855 nodeName:}" failed. No retries permitted until 2026-09-16 18:56:29.016584009 +0000 UTC m=+690.320600953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert") pod "odh-model-controller-7b966d7984-nv2qk" (UID: "4ca058e9-7f14-41fb-b0d3-3ba5d19b0855") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:56:28.518738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.518714    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ef3166-3d20-4dd4-85d4-3b10cbc00bfd-tls-certs\") pod \"model-serving-api-7b954ffc95-6kmjf\" (UID: \"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd\") " pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.525478 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.525453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv77\" (UniqueName: \"kubernetes.io/projected/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-kube-api-access-4jv77\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:28.525582 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.525547    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzkkc\" (UniqueName: \"kubernetes.io/projected/86ef3166-3d20-4dd4-85d4-3b10cbc00bfd-kube-api-access-zzkkc\") pod \"model-serving-api-7b954ffc95-6kmjf\" (UID: \"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd\") " pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.601533 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.601458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:28.871410 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:28.871384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-6kmjf"]
Sep 16 18:56:28.873135 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:56:28.873104    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ef3166_3d20_4dd4_85d4_3b10cbc00bfd.slice/crio-cad7276eb0d14f029ba68e443b955ebedafb26beed737ecec4f26a01ca5377d4 WatchSource:0}: Error finding container cad7276eb0d14f029ba68e443b955ebedafb26beed737ecec4f26a01ca5377d4: Status 404 returned error can't find the container with id cad7276eb0d14f029ba68e443b955ebedafb26beed737ecec4f26a01ca5377d4
Sep 16 18:56:29.021797 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:29.021761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-metrics-certs\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:29.021977 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:29.021807    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:29.021977 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:56:29.021947    2581 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 18:56:29.022067 ip-10-0-133-194 kubenswrapper[2581]: E0916 18:56:29.021996    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert podName:4ca058e9-7f14-41fb-b0d3-3ba5d19b0855 nodeName:}" failed. No retries permitted until 2026-09-16 18:56:30.021984255 +0000 UTC m=+691.326001200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert") pod "odh-model-controller-7b966d7984-nv2qk" (UID: "4ca058e9-7f14-41fb-b0d3-3ba5d19b0855") : secret "odh-model-controller-webhook-cert" not found
Sep 16 18:56:29.024194 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:29.024171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-metrics-certs\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:29.051758 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:29.051731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-6kmjf" event={"ID":"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd","Type":"ContainerStarted","Data":"cad7276eb0d14f029ba68e443b955ebedafb26beed737ecec4f26a01ca5377d4"}
Sep 16 18:56:30.029491 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:30.029451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:30.032602 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:30.032578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ca058e9-7f14-41fb-b0d3-3ba5d19b0855-cert\") pod \"odh-model-controller-7b966d7984-nv2qk\" (UID: \"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855\") " pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:30.115218 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:30.115186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:31.346045 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:31.346021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-nv2qk"]
Sep 16 18:56:31.347692 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:56:31.347666    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca058e9_7f14_41fb_b0d3_3ba5d19b0855.slice/crio-7f5a8a221f6f2ca21bb5eaff419511402547677f0e87f8ca75a4e4bd20f94230 WatchSource:0}: Error finding container 7f5a8a221f6f2ca21bb5eaff419511402547677f0e87f8ca75a4e4bd20f94230: Status 404 returned error can't find the container with id 7f5a8a221f6f2ca21bb5eaff419511402547677f0e87f8ca75a4e4bd20f94230
Sep 16 18:56:32.069337 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:32.069288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-6kmjf" event={"ID":"86ef3166-3d20-4dd4-85d4-3b10cbc00bfd","Type":"ContainerStarted","Data":"4ab0ba4d842c222aaafac08593ed51173fa42779e1a91043d403dd2ff4588381"}
Sep 16 18:56:32.069535 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:32.069409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:32.070630 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:32.070599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nv2qk" event={"ID":"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855","Type":"ContainerStarted","Data":"7f5a8a221f6f2ca21bb5eaff419511402547677f0e87f8ca75a4e4bd20f94230"}
Sep 16 18:56:32.089050 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:32.088992    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-6kmjf" podStartSLOduration=1.829713825 podStartE2EDuration="4.088972779s" totalImagesPullingTime="2.259258954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:32.086493842 +0000 UTC m=+693.390510810" watchObservedRunningTime="2026-09-16 18:56:32.088972779 +0000 UTC m=+693.392989747"
Sep 16 18:56:34.082249 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:34.082219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-nv2qk" event={"ID":"4ca058e9-7f14-41fb-b0d3-3ba5d19b0855","Type":"ContainerStarted","Data":"ad1601e6be0885e4e69daaa101ace42a3cdd6099745a12fe0e46aff400f6ac12"}
Sep 16 18:56:34.082563 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:34.082279    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:34.100537 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:34.100492    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-nv2qk" podStartSLOduration=3.441021322 podStartE2EDuration="6.100477699s" totalImagesPullingTime="2.659456377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:34.099125292 +0000 UTC m=+695.403142260" watchObservedRunningTime="2026-09-16 18:56:34.100477699 +0000 UTC m=+695.404494692"
Sep 16 18:56:43.081036 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:43.081006    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-6kmjf"
Sep 16 18:56:45.090519 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:45.090492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-nv2qk"
Sep 16 18:56:45.910506 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:45.910470    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-cxp59"]
Sep 16 18:56:45.913809 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:45.913792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cxp59"
Sep 16 18:56:45.916357 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:45.916338    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:56:45.922789 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:45.922765    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cxp59"]
Sep 16 18:56:46.059848 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.059816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-model-cache\") pod \"s3-init-cxp59\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") " pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.060044 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.059924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdmhb\" (UniqueName: \"kubernetes.io/projected/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-kube-api-access-tdmhb\") pod \"s3-init-cxp59\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") " pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.161097 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.161011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-model-cache\") pod \"s3-init-cxp59\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") " pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.161469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.161118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdmhb\" (UniqueName: \"kubernetes.io/projected/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-kube-api-access-tdmhb\") pod \"s3-init-cxp59\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") " pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.161469 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.161438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-model-cache\") pod \"s3-init-cxp59\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") " pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.169571 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.169552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdmhb\" (UniqueName: \"kubernetes.io/projected/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-kube-api-access-tdmhb\") pod \"s3-init-cxp59\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") " pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.223788 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.223765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cxp59"
Sep 16 18:56:46.497138 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:46.497096    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-cxp59"]
Sep 16 18:56:46.499252 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:56:46.499214    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58139ed6_7b53_4870_aa73_6bbf9cfb33e1.slice/crio-78e6c226f8235d42a3c55c1d31da6233b49c04069e4c1ca6c024ea3f5208cef9 WatchSource:0}: Error finding container 78e6c226f8235d42a3c55c1d31da6233b49c04069e4c1ca6c024ea3f5208cef9: Status 404 returned error can't find the container with id 78e6c226f8235d42a3c55c1d31da6233b49c04069e4c1ca6c024ea3f5208cef9
Sep 16 18:56:47.147160 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:47.147122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cxp59" event={"ID":"58139ed6-7b53-4870-aa73-6bbf9cfb33e1","Type":"ContainerStarted","Data":"78e6c226f8235d42a3c55c1d31da6233b49c04069e4c1ca6c024ea3f5208cef9"}
Sep 16 18:56:51.170504 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:51.170467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cxp59" event={"ID":"58139ed6-7b53-4870-aa73-6bbf9cfb33e1","Type":"ContainerStarted","Data":"5f92b159b86e356c2346b146b05e0d6737add2ad7168bcc9a7d1220f949dd903"}
Sep 16 18:56:52.177759 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:52.177726    2581 generic.go:356] "Generic (PLEG): container finished" podID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerID="5f92b159b86e356c2346b146b05e0d6737add2ad7168bcc9a7d1220f949dd903" exitCode=0
Sep 16 18:56:52.178122 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:52.177774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cxp59" event={"ID":"58139ed6-7b53-4870-aa73-6bbf9cfb33e1","Type":"ContainerDied","Data":"5f92b159b86e356c2346b146b05e0d6737add2ad7168bcc9a7d1220f949dd903"}
Sep 16 18:56:57.206833 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:57.206793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cxp59" event={"ID":"58139ed6-7b53-4870-aa73-6bbf9cfb33e1","Type":"ContainerStarted","Data":"6aedbf78031ec78b23503b51018d2e36dbb701895cdf1f28d42a992991f55379"}
Sep 16 18:56:57.224322 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:56:57.224266    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-cxp59" podStartSLOduration=2.3616908629999998 podStartE2EDuration="12.224247655s" totalImagesPullingTime="8.244103742s" totalInitContainerRuntime="1.61845305s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:57.224054324 +0000 UTC m=+718.528071292" watchObservedRunningTime="2026-09-16 18:56:57.224247655 +0000 UTC m=+718.528264624"
Sep 16 18:58:01.523655 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:01.523622    2581 generic.go:356] "Generic (PLEG): container finished" podID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerID="6aedbf78031ec78b23503b51018d2e36dbb701895cdf1f28d42a992991f55379" exitCode=0
Sep 16 18:58:01.524030 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:01.523693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cxp59" event={"ID":"58139ed6-7b53-4870-aa73-6bbf9cfb33e1","Type":"ContainerDied","Data":"6aedbf78031ec78b23503b51018d2e36dbb701895cdf1f28d42a992991f55379"}
Sep 16 18:58:02.656085 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.656056    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cxp59"
Sep 16 18:58:02.765584 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.765554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-model-cache\") pod \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") "
Sep 16 18:58:02.765743 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.765607    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdmhb\" (UniqueName: \"kubernetes.io/projected/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-kube-api-access-tdmhb\") pod \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\" (UID: \"58139ed6-7b53-4870-aa73-6bbf9cfb33e1\") "
Sep 16 18:58:02.767635 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.767606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-kube-api-access-tdmhb" (OuterVolumeSpecName: "kube-api-access-tdmhb") pod "58139ed6-7b53-4870-aa73-6bbf9cfb33e1" (UID: "58139ed6-7b53-4870-aa73-6bbf9cfb33e1"). InnerVolumeSpecName "kube-api-access-tdmhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:02.787032 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.786983    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "58139ed6-7b53-4870-aa73-6bbf9cfb33e1" (UID: "58139ed6-7b53-4870-aa73-6bbf9cfb33e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:02.866201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.866170    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:58:02.866201 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:02.866196    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdmhb\" (UniqueName: \"kubernetes.io/projected/58139ed6-7b53-4870-aa73-6bbf9cfb33e1-kube-api-access-tdmhb\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 18:58:03.534378 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:03.534303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-cxp59" event={"ID":"58139ed6-7b53-4870-aa73-6bbf9cfb33e1","Type":"ContainerDied","Data":"78e6c226f8235d42a3c55c1d31da6233b49c04069e4c1ca6c024ea3f5208cef9"}
Sep 16 18:58:03.534378 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:03.534340    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78e6c226f8235d42a3c55c1d31da6233b49c04069e4c1ca6c024ea3f5208cef9"
Sep 16 18:58:03.534378 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:03.534308    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-cxp59"
Sep 16 18:58:38.738881 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.738831    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:58:38.739282 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.739175    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerName="s3-init"
Sep 16 18:58:38.739282 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.739185    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerName="s3-init"
Sep 16 18:58:38.739282 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.739193    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerName="download-hf-model"
Sep 16 18:58:38.739282 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.739198    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerName="download-hf-model"
Sep 16 18:58:38.739282 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.739257    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58139ed6-7b53-4870-aa73-6bbf9cfb33e1" containerName="s3-init"
Sep 16 18:58:38.742183 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.742166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.745179 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.745151    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:58:38.745394 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.745375    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:58:38.745485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.745404    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:58:38.745485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.745450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:58:38.745485 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.745408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 18:58:38.746584 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.746566    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-5nx7z\""
Sep 16 18:58:38.746679 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.746587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-ckqk4\""
Sep 16 18:58:38.759907 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.759884    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:58:38.840554 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55ad273-fd07-497a-8796-16932611dab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840554 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840747 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgc4z\" (UniqueName: \"kubernetes.io/projected/e55ad273-fd07-497a-8796-16932611dab1-kube-api-access-xgc4z\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840747 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840747 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840677    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840747 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e55ad273-fd07-497a-8796-16932611dab1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.840902 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.840799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941373 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55ad273-fd07-497a-8796-16932611dab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941373 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgc4z\" (UniqueName: \"kubernetes.io/projected/e55ad273-fd07-497a-8796-16932611dab1-kube-api-access-xgc4z\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e55ad273-fd07-497a-8796-16932611dab1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941605 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941966 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.941966 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.942065 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.941995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.942065 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.942006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.942169 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.942115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e55ad273-fd07-497a-8796-16932611dab1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.943992 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.943960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.944117 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.944076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55ad273-fd07-497a-8796-16932611dab1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.948533 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.948513    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"]
Sep 16 18:58:38.952204 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.952182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:38.955222 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.955194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-wnqtr\""
Sep 16 18:58:38.956590 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.956565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgc4z\" (UniqueName: \"kubernetes.io/projected/e55ad273-fd07-497a-8796-16932611dab1-kube-api-access-xgc4z\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:38.963103 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:38.963084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"]
Sep 16 18:58:39.042265 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.042176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjczd\" (UniqueName: \"kubernetes.io/projected/d4392bf5-9951-4576-837d-a97043d14549-kube-api-access-xjczd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.042406 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.042263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4392bf5-9951-4576-837d-a97043d14549-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.052158 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.052133    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:58:39.143550 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.143516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4392bf5-9951-4576-837d-a97043d14549-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.143682 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.143634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjczd\" (UniqueName: \"kubernetes.io/projected/d4392bf5-9951-4576-837d-a97043d14549-kube-api-access-xjczd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.145806 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.145780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4392bf5-9951-4576-837d-a97043d14549-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.151444 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.151419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjczd\" (UniqueName: \"kubernetes.io/projected/d4392bf5-9951-4576-837d-a97043d14549-kube-api-access-xjczd\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.275271 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.275239    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:39.331668 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.331644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 18:58:39.334463 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:58:39.334435    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55ad273_fd07_497a_8796_16932611dab1.slice/crio-0929d157e973d281f66f600b4c7a92be4defe6e358981d640e96576c836c0a0b WatchSource:0}: Error finding container 0929d157e973d281f66f600b4c7a92be4defe6e358981d640e96576c836c0a0b: Status 404 returned error can't find the container with id 0929d157e973d281f66f600b4c7a92be4defe6e358981d640e96576c836c0a0b
Sep 16 18:58:39.563226 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.563202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"]
Sep 16 18:58:39.565844 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:58:39.565812    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4392bf5_9951_4576_837d_a97043d14549.slice/crio-883cef377e67c48e292f7de46ce1d974fe88d75479da0629078646db398b7db9 WatchSource:0}: Error finding container 883cef377e67c48e292f7de46ce1d974fe88d75479da0629078646db398b7db9: Status 404 returned error can't find the container with id 883cef377e67c48e292f7de46ce1d974fe88d75479da0629078646db398b7db9
Sep 16 18:58:39.710295 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.710253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e55ad273-fd07-497a-8796-16932611dab1","Type":"ContainerStarted","Data":"111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12"}
Sep 16 18:58:39.710454 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.710324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e55ad273-fd07-497a-8796-16932611dab1","Type":"ContainerStarted","Data":"0929d157e973d281f66f600b4c7a92be4defe6e358981d640e96576c836c0a0b"}
Sep 16 18:58:39.711440 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:39.711415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" event={"ID":"d4392bf5-9951-4576-837d-a97043d14549","Type":"ContainerStarted","Data":"883cef377e67c48e292f7de46ce1d974fe88d75479da0629078646db398b7db9"}
Sep 16 18:58:42.739245 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:42.739206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" event={"ID":"d4392bf5-9951-4576-837d-a97043d14549","Type":"ContainerStarted","Data":"8aa7053d92a446c6af4cc4e93aa6465576f932b752fc623b34595722bb6312bc"}
Sep 16 18:58:42.739668 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:42.739377    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:58:42.749666 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:42.749624    2581 generic.go:356] "Generic (PLEG): container finished" podID="e55ad273-fd07-497a-8796-16932611dab1" containerID="111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12" exitCode=0
Sep 16 18:58:42.749825 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:42.749694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e55ad273-fd07-497a-8796-16932611dab1","Type":"ContainerDied","Data":"111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12"}
Sep 16 18:58:42.759840 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:42.759795    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podStartSLOduration=1.7257464580000001 podStartE2EDuration="4.759778511s" totalImagesPullingTime="3.034032053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:42.756363498 +0000 UTC m=+824.060380468" watchObservedRunningTime="2026-09-16 18:58:42.759778511 +0000 UTC m=+824.063795479"
Sep 16 18:58:49.827939 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:49.827904    2581 generic.go:356] "Generic (PLEG): container finished" podID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerID="d7dfcd134816e676cb3e11312f0f8996019b70f5720dc50289f4e9d3421225df" exitCode=0
Sep 16 18:58:49.828405 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:49.827984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" event={"ID":"9c6de2c0-b034-4e27-8331-8113e71873a5","Type":"ContainerDied","Data":"d7dfcd134816e676cb3e11312f0f8996019b70f5720dc50289f4e9d3421225df"}
Sep 16 18:58:49.828521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:49.828505    2581 scope.go:117] "RemoveContainer" containerID="d7dfcd134816e676cb3e11312f0f8996019b70f5720dc50289f4e9d3421225df"
Sep 16 18:58:49.829599 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:49.829561    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:58:49.829695 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:49.829651    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:58:49.829742 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:49.829697    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 18:58:51.484884 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.484536    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"]
Sep 16 18:58:51.495795 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.495765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.498815 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.498788    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:58:51.500405 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.500382    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-tdxrk\""
Sep 16 18:58:51.500526 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.500406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"]
Sep 16 18:58:51.500647 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.500627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:58:51.501660 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.501514    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:58:51.568837 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.568805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-dshm\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.569024 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.568941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6w78\" (UniqueName: \"kubernetes.io/projected/99dc2db0-1031-45bb-b58c-2c1716240220-kube-api-access-t6w78\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.569024 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.568980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.569024 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.569020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99dc2db0-1031-45bb-b58c-2c1716240220-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.569247 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.569054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-home\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.569247 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.569165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.670738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.670738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99dc2db0-1031-45bb-b58c-2c1716240220-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.670738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-home\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.670738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.670738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-dshm\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.670738 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6w78\" (UniqueName: \"kubernetes.io/projected/99dc2db0-1031-45bb-b58c-2c1716240220-kube-api-access-t6w78\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.672521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.670996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.672521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.671304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.672521 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.671613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-home\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.673381 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.673335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-dshm\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.675167 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.675116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99dc2db0-1031-45bb-b58c-2c1716240220-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.685244 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.684987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6w78\" (UniqueName: \"kubernetes.io/projected/99dc2db0-1031-45bb-b58c-2c1716240220-kube-api-access-t6w78\") pod \"fc-smoke-test-kserve-7f4687897-wvzz4\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:51.812200 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:51.812040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:58:55.876631 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:55.876590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" event={"ID":"9c6de2c0-b034-4e27-8331-8113e71873a5","Type":"ContainerStarted","Data":"9e87cc22be167166ad674b497db96ef5dfc6fed78ed9c27b61554a0cb0bfe190"}
Sep 16 18:58:56.121279 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:56.121248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"]
Sep 16 18:58:56.122925 ip-10-0-133-194 kubenswrapper[2581]: W0916 18:58:56.122894    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dc2db0_1031_45bb_b58c_2c1716240220.slice/crio-6f637a8cd091171068680de3c88818e95a6e72d59504a826485049712519cad6 WatchSource:0}: Error finding container 6f637a8cd091171068680de3c88818e95a6e72d59504a826485049712519cad6: Status 404 returned error can't find the container with id 6f637a8cd091171068680de3c88818e95a6e72d59504a826485049712519cad6
Sep 16 18:58:56.776073 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:56.775987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:58:56.777344 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:56.777314    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 16 18:58:56.777472 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:56.777372    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podUID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 16 18:58:56.884538 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:56.884463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" event={"ID":"99dc2db0-1031-45bb-b58c-2c1716240220","Type":"ContainerStarted","Data":"6f637a8cd091171068680de3c88818e95a6e72d59504a826485049712519cad6"}
Sep 16 18:58:57.775844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:57.775725    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 16 18:58:57.775844 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:57.775793    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podUID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 16 18:58:58.013518 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:58.013485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:58:58.776059 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:58.775837    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 16 18:58:58.776059 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:58.775926    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podUID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 16 18:58:59.776178 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:59.776133    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 16 18:58:59.776611 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:58:59.776219    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podUID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 16 18:59:00.775759 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:00.775651    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 16 18:59:00.775759 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:00.775745    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podUID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 16 18:59:01.776554 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:01.776522    2581 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body=
Sep 16 18:59:01.776939 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:01.776589    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm" podUID="9c6de2c0-b034-4e27-8331-8113e71873a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused"
Sep 16 18:59:02.779996 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:02.779956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:59:02.781288 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:02.781259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-5p4nm"
Sep 16 18:59:13.775374 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:13.775343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 18:59:14.989560 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:14.989515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" event={"ID":"99dc2db0-1031-45bb-b58c-2c1716240220","Type":"ContainerStarted","Data":"684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"}
Sep 16 18:59:14.991443 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:14.991411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e55ad273-fd07-497a-8796-16932611dab1","Type":"ContainerStarted","Data":"0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5"}
Sep 16 18:59:15.010035 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:15.009991    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" podStartSLOduration=5.883414812 podStartE2EDuration="24.009976409s" totalImagesPullingTime="18.126561597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:15.009869637 +0000 UTC m=+856.313886601" watchObservedRunningTime="2026-09-16 18:59:15.009976409 +0000 UTC m=+856.313993385"
Sep 16 18:59:15.031734 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:15.031681    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.760879779 podStartE2EDuration="37.03166473s" totalImagesPullingTime="31.5591293s" totalInitContainerRuntime="2.711655651s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:15.031562691 +0000 UTC m=+856.335579659" watchObservedRunningTime="2026-09-16 18:59:15.03166473 +0000 UTC m=+856.335681702"
Sep 16 18:59:15.052721 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:15.052693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:59:15.812676 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:15.812640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:59:19.052962 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:19.052920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 18:59:19.054729 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:19.054699    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 18:59:21.812550 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:21.812509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:59:21.825795 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:21.825760    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:59:22.037610 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:22.037571    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 18:59:29.052800 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:29.052751    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 18:59:39.053322 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:39.053280    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 18:59:49.053481 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:49.053432    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 18:59:59.053301 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:59.053205    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 18:59:59.186739 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:59.186705    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:59:59.187653 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:59.187623    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 18:59:59.189542 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:59.189521    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 18:59:59.190437 ip-10-0-133-194 kubenswrapper[2581]: I0916 18:59:59.190420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:00:02.049818 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.049780    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"]
Sep 16 19:00:02.068413 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:00:02.068368    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:00:02.068413 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:02.068413 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:02.068413 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219" cmd=["/bin/sleep","15"]
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.068440    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" message="" err=<
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:00:02.068453    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" podUID="99dc2db0-1031-45bb-b58c-2c1716240220" containerName="main" containerID="cri-o://684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"
Sep 16 19:00:02.068656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.068502    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" podUID="99dc2db0-1031-45bb-b58c-2c1716240220" containerName="main" containerID="cri-o://684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219" gracePeriod=60
Sep 16 19:00:02.421778 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.421747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 19:00:02.424976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.424956    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-home\") pod \"99dc2db0-1031-45bb-b58c-2c1716240220\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") "
Sep 16 19:00:02.425091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425004    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99dc2db0-1031-45bb-b58c-2c1716240220-tls-certs\") pod \"99dc2db0-1031-45bb-b58c-2c1716240220\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") "
Sep 16 19:00:02.425091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425044    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-model-cache\") pod \"99dc2db0-1031-45bb-b58c-2c1716240220\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") "
Sep 16 19:00:02.425091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-tmp-dir\") pod \"99dc2db0-1031-45bb-b58c-2c1716240220\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") "
Sep 16 19:00:02.425251 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6w78\" (UniqueName: \"kubernetes.io/projected/99dc2db0-1031-45bb-b58c-2c1716240220-kube-api-access-t6w78\") pod \"99dc2db0-1031-45bb-b58c-2c1716240220\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") "
Sep 16 19:00:02.425251 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-dshm\") pod \"99dc2db0-1031-45bb-b58c-2c1716240220\" (UID: \"99dc2db0-1031-45bb-b58c-2c1716240220\") "
Sep 16 19:00:02.425251 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-home" (OuterVolumeSpecName: "home") pod "99dc2db0-1031-45bb-b58c-2c1716240220" (UID: "99dc2db0-1031-45bb-b58c-2c1716240220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:02.425427 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99dc2db0-1031-45bb-b58c-2c1716240220" (UID: "99dc2db0-1031-45bb-b58c-2c1716240220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:02.425492 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425412    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:00:02.425546 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.425486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-model-cache" (OuterVolumeSpecName: "model-cache") pod "99dc2db0-1031-45bb-b58c-2c1716240220" (UID: "99dc2db0-1031-45bb-b58c-2c1716240220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:02.427313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.427284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99dc2db0-1031-45bb-b58c-2c1716240220-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99dc2db0-1031-45bb-b58c-2c1716240220" (UID: "99dc2db0-1031-45bb-b58c-2c1716240220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:02.427752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.427731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-dshm" (OuterVolumeSpecName: "dshm") pod "99dc2db0-1031-45bb-b58c-2c1716240220" (UID: "99dc2db0-1031-45bb-b58c-2c1716240220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:02.427949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.427928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99dc2db0-1031-45bb-b58c-2c1716240220-kube-api-access-t6w78" (OuterVolumeSpecName: "kube-api-access-t6w78") pod "99dc2db0-1031-45bb-b58c-2c1716240220" (UID: "99dc2db0-1031-45bb-b58c-2c1716240220"). InnerVolumeSpecName "kube-api-access-t6w78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:02.525777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.525745    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99dc2db0-1031-45bb-b58c-2c1716240220-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:00:02.525777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.525772    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:00:02.525777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.525781    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:00:02.526020 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.525790    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6w78\" (UniqueName: \"kubernetes.io/projected/99dc2db0-1031-45bb-b58c-2c1716240220-kube-api-access-t6w78\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:00:02.526020 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:02.525799    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99dc2db0-1031-45bb-b58c-2c1716240220-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:00:03.253210 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.253176    2581 generic.go:356] "Generic (PLEG): container finished" podID="99dc2db0-1031-45bb-b58c-2c1716240220" containerID="684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219" exitCode=0
Sep 16 19:00:03.253611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.253223    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"
Sep 16 19:00:03.253611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.253242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" event={"ID":"99dc2db0-1031-45bb-b58c-2c1716240220","Type":"ContainerDied","Data":"684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"}
Sep 16 19:00:03.253611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.253296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4" event={"ID":"99dc2db0-1031-45bb-b58c-2c1716240220","Type":"ContainerDied","Data":"6f637a8cd091171068680de3c88818e95a6e72d59504a826485049712519cad6"}
Sep 16 19:00:03.253611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.253334    2581 scope.go:117] "RemoveContainer" containerID="684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"
Sep 16 19:00:03.262820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.262799    2581 scope.go:117] "RemoveContainer" containerID="684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"
Sep 16 19:00:03.263141 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:00:03.263121    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219\": container with ID starting with 684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219 not found: ID does not exist" containerID="684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"
Sep 16 19:00:03.263217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.263149    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219"} err="failed to get container status \"684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219\": rpc error: code = NotFound desc = could not find container \"684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219\": container with ID starting with 684b76cba20cd98cf25f035fbb1236220fa54155e589712083cc534e3f33e219 not found: ID does not exist"
Sep 16 19:00:03.270889 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.270849    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"]
Sep 16 19:00:03.272890 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:03.272869    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-wvzz4"]
Sep 16 19:00:05.222340 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:05.222306    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99dc2db0-1031-45bb-b58c-2c1716240220" path="/var/lib/kubelet/pods/99dc2db0-1031-45bb-b58c-2c1716240220/volumes"
Sep 16 19:00:09.053614 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:09.053575    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 19:00:19.053595 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:19.053554    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 19:00:29.052659 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:29.052611    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 19:00:39.052882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:39.052808    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 19:00:49.062137 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:49.062105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:00:49.069954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:00:49.069928    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:01:08.177606 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.177574    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"]
Sep 16 19:01:08.177999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.177904    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99dc2db0-1031-45bb-b58c-2c1716240220" containerName="main"
Sep 16 19:01:08.177999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.177917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99dc2db0-1031-45bb-b58c-2c1716240220" containerName="main"
Sep 16 19:01:08.177999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.177979    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99dc2db0-1031-45bb-b58c-2c1716240220" containerName="main"
Sep 16 19:01:08.182134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.182117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.187534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.187512    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-n2j5t\""
Sep 16 19:01:08.187645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.187529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 19:01:08.189047 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.189031    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 19:01:08.190049 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.190033    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 19:01:08.203747 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.203722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"]
Sep 16 19:01:08.360419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.360381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.360419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.360419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.360641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.360463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt26w\" (UniqueName: \"kubernetes.io/projected/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-kube-api-access-nt26w\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.360641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.360508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.360641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.360568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.360641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.360626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461381 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461381 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461381 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt26w\" (UniqueName: \"kubernetes.io/projected/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-kube-api-access-nt26w\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-home\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.461804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.461750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.462186 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.462157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.463711 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.463681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.463905 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.463885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.475164 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.475143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt26w\" (UniqueName: \"kubernetes.io/projected/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-kube-api-access-nt26w\") pod \"fc-concurrency-test-kserve-b5d44467c-9l89x\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.492040 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.492015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:08.781614 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:08.781593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"]
Sep 16 19:01:08.783380 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:01:08.783351    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30d9e9e_10ed_44bf_b442_e1c0dab64c2b.slice/crio-4e2234817cb6c0fe7b421a497a59e93326f5b1f6db78f5b831b53874f4cb4c9e WatchSource:0}: Error finding container 4e2234817cb6c0fe7b421a497a59e93326f5b1f6db78f5b831b53874f4cb4c9e: Status 404 returned error can't find the container with id 4e2234817cb6c0fe7b421a497a59e93326f5b1f6db78f5b831b53874f4cb4c9e
Sep 16 19:01:09.603633 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:09.603599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" event={"ID":"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b","Type":"ContainerStarted","Data":"1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"}
Sep 16 19:01:09.604072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:09.603650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" event={"ID":"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b","Type":"ContainerStarted","Data":"4e2234817cb6c0fe7b421a497a59e93326f5b1f6db78f5b831b53874f4cb4c9e"}
Sep 16 19:01:09.646606 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:09.646547    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" podStartSLOduration=1.64652952 podStartE2EDuration="1.64652952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:09.64598741 +0000 UTC m=+970.950004379" watchObservedRunningTime="2026-09-16 19:01:09.64652952 +0000 UTC m=+970.950546516"
Sep 16 19:01:10.492137 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:10.492079    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:18.492600 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:18.492561    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:18.505040 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:18.505014    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:18.661069 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:18.661035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:19.681654 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:19.681609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:01:19.698399 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:19.698366    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"]
Sep 16 19:01:23.774919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:23.774817    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:33.774838 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:33.774797    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:34.705592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:34.705529    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" containerID="cri-o://0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5" gracePeriod=45
Sep 16 19:01:34.716809 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:34.716774    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main" containerID="cri-o://8aa7053d92a446c6af4cc4e93aa6465576f932b752fc623b34595722bb6312bc" gracePeriod=45
Sep 16 19:01:35.063299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.063188    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Sep 16 19:01:35.453307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.453280    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:01:35.596297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-tmp-dir\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596268    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-home\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596306    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgc4z\" (UniqueName: \"kubernetes.io/projected/e55ad273-fd07-497a-8796-16932611dab1-kube-api-access-xgc4z\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-model-cache\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-dshm\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55ad273-fd07-497a-8796-16932611dab1-tls-certs\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-kserve-provision-location\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596492    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e55ad273-fd07-497a-8796-16932611dab1-cabundle-cert\") pod \"e55ad273-fd07-497a-8796-16932611dab1\" (UID: \"e55ad273-fd07-497a-8796-16932611dab1\") "
Sep 16 19:01:35.596802 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:35.597007 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-home" (OuterVolumeSpecName: "home") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:35.597100 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.596984    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55ad273-fd07-497a-8796-16932611dab1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:35.598791 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.598766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-dshm" (OuterVolumeSpecName: "dshm") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:35.598938 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.598917    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55ad273-fd07-497a-8796-16932611dab1-kube-api-access-xgc4z" (OuterVolumeSpecName: "kube-api-access-xgc4z") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "kube-api-access-xgc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:35.599004 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.598933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55ad273-fd07-497a-8796-16932611dab1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:35.615333 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.615302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:35.617180 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.617156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e55ad273-fd07-497a-8796-16932611dab1" (UID: "e55ad273-fd07-497a-8796-16932611dab1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:35.697397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697371    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697394    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697403    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgc4z\" (UniqueName: \"kubernetes.io/projected/e55ad273-fd07-497a-8796-16932611dab1-kube-api-access-xgc4z\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697411    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697420    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697427    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e55ad273-fd07-497a-8796-16932611dab1-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697434    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e55ad273-fd07-497a-8796-16932611dab1-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.697534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.697442    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e55ad273-fd07-497a-8796-16932611dab1-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:35.740226 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.740198    2581 generic.go:356] "Generic (PLEG): container finished" podID="e55ad273-fd07-497a-8796-16932611dab1" containerID="0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5" exitCode=0
Sep 16 19:01:35.740313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.740238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e55ad273-fd07-497a-8796-16932611dab1","Type":"ContainerDied","Data":"0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5"}
Sep 16 19:01:35.740313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.740291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"e55ad273-fd07-497a-8796-16932611dab1","Type":"ContainerDied","Data":"0929d157e973d281f66f600b4c7a92be4defe6e358981d640e96576c836c0a0b"}
Sep 16 19:01:35.740404 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.740314    2581 scope.go:117] "RemoveContainer" containerID="0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5"
Sep 16 19:01:35.740404 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.740246    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 16 19:01:35.749414 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.749398    2581 scope.go:117] "RemoveContainer" containerID="111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12"
Sep 16 19:01:35.756698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.756682    2581 scope.go:117] "RemoveContainer" containerID="0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5"
Sep 16 19:01:35.756973 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:01:35.756956    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5\": container with ID starting with 0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5 not found: ID does not exist" containerID="0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5"
Sep 16 19:01:35.757025 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.756980    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5"} err="failed to get container status \"0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5\": rpc error: code = NotFound desc = could not find container \"0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5\": container with ID starting with 0395b34796e717fff9fc6ba73b9896708e02b3d760c1425855fb2797b91426c5 not found: ID does not exist"
Sep 16 19:01:35.757025 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.756997    2581 scope.go:117] "RemoveContainer" containerID="111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12"
Sep 16 19:01:35.757208 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:01:35.757192    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12\": container with ID starting with 111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12 not found: ID does not exist" containerID="111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12"
Sep 16 19:01:35.757248 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.757214    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12"} err="failed to get container status \"111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12\": rpc error: code = NotFound desc = could not find container \"111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12\": container with ID starting with 111fd3d28129a7586f5dc96fd10aec17b8bcbc10d449394780f2042faafdcc12 not found: ID does not exist"
Sep 16 19:01:35.765592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.765572    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:01:35.774229 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:35.774205    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 16 19:01:37.221954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:37.221921    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55ad273-fd07-497a-8796-16932611dab1" path="/var/lib/kubelet/pods/e55ad273-fd07-497a-8796-16932611dab1/volumes"
Sep 16 19:01:43.775052 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:43.775015    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:43.775535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:43.775131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 19:01:47.603301 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:47.603265    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"]
Sep 16 19:01:47.613244 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:01:47.613203    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:01:47.613244 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:47.613244 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:47.613244 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e" cmd=["/bin/sleep","15"]
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:47.613256    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" message="" err=<
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:01:47.613270    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerName="main" containerID="cri-o://1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"
Sep 16 19:01:47.613516 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:47.613306    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerName="main" containerID="cri-o://1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e" gracePeriod=60
Sep 16 19:01:47.650048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:47.650018    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": dial tcp 10.132.0.39:8000: connect: connection refused"
Sep 16 19:01:47.961800 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:47.961777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:48.099825 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.099796    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tmp-dir\") pod \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") "
Sep 16 19:01:48.100029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.099906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt26w\" (UniqueName: \"kubernetes.io/projected/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-kube-api-access-nt26w\") pod \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") "
Sep 16 19:01:48.100029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.099941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-dshm\") pod \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") "
Sep 16 19:01:48.100029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.099973    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-home\") pod \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") "
Sep 16 19:01:48.100029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100000    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tls-certs\") pod \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") "
Sep 16 19:01:48.100029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-model-cache\") pod \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\" (UID: \"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b\") "
Sep 16 19:01:48.100288 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" (UID: "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:48.100288 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-home" (OuterVolumeSpecName: "home") pod "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" (UID: "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:48.100375 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100309    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.100375 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" (UID: "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:48.100375 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.100326    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.102235 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.102213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-dshm" (OuterVolumeSpecName: "dshm") pod "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" (UID: "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:48.102312 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.102264    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" (UID: "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:48.102353 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.102328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-kube-api-access-nt26w" (OuterVolumeSpecName: "kube-api-access-nt26w") pod "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" (UID: "c30d9e9e-10ed-44bf-b442-e1c0dab64c2b"). InnerVolumeSpecName "kube-api-access-nt26w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:48.200846 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.200819    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.200846 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.200842    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.201008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.200874    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.201008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.200888    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt26w\" (UniqueName: \"kubernetes.io/projected/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b-kube-api-access-nt26w\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:01:48.805813 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.805774    2581 generic.go:356] "Generic (PLEG): container finished" podID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerID="1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e" exitCode=0
Sep 16 19:01:48.806206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.805838    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"
Sep 16 19:01:48.806206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.805840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" event={"ID":"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b","Type":"ContainerDied","Data":"1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"}
Sep 16 19:01:48.806206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.805945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x" event={"ID":"c30d9e9e-10ed-44bf-b442-e1c0dab64c2b","Type":"ContainerDied","Data":"4e2234817cb6c0fe7b421a497a59e93326f5b1f6db78f5b831b53874f4cb4c9e"}
Sep 16 19:01:48.806206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.805975    2581 scope.go:117] "RemoveContainer" containerID="1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"
Sep 16 19:01:48.815494 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.815474    2581 scope.go:117] "RemoveContainer" containerID="1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"
Sep 16 19:01:48.815751 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:01:48.815731    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e\": container with ID starting with 1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e not found: ID does not exist" containerID="1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"
Sep 16 19:01:48.815814 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.815763    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e"} err="failed to get container status \"1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e\": rpc error: code = NotFound desc = could not find container \"1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e\": container with ID starting with 1b81446c9b3a1978c7d5dab5f0c87ecb96f98b5a71139c14fd1d9001a0bfaf0e not found: ID does not exist"
Sep 16 19:01:48.831539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.831517    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"]
Sep 16 19:01:48.840945 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:48.840923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9l89x"]
Sep 16 19:01:49.222065 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:49.222033    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" path="/var/lib/kubelet/pods/c30d9e9e-10ed-44bf-b442-e1c0dab64c2b/volumes"
Sep 16 19:01:53.775350 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:01:53.775314    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:03.775228 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:03.775190    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:04.886531 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:04.886502    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4392bf5-9951-4576-837d-a97043d14549" containerID="8aa7053d92a446c6af4cc4e93aa6465576f932b752fc623b34595722bb6312bc" exitCode=0
Sep 16 19:02:04.886881 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:04.886566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" event={"ID":"d4392bf5-9951-4576-837d-a97043d14549","Type":"ContainerDied","Data":"8aa7053d92a446c6af4cc4e93aa6465576f932b752fc623b34595722bb6312bc"}
Sep 16 19:02:04.970514 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:04.970492    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 19:02:05.131394 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.131285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4392bf5-9951-4576-837d-a97043d14549-tls-certs\") pod \"d4392bf5-9951-4576-837d-a97043d14549\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") "
Sep 16 19:02:05.131394 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.131387    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjczd\" (UniqueName: \"kubernetes.io/projected/d4392bf5-9951-4576-837d-a97043d14549-kube-api-access-xjczd\") pod \"d4392bf5-9951-4576-837d-a97043d14549\" (UID: \"d4392bf5-9951-4576-837d-a97043d14549\") "
Sep 16 19:02:05.133408 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.133383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4392bf5-9951-4576-837d-a97043d14549-kube-api-access-xjczd" (OuterVolumeSpecName: "kube-api-access-xjczd") pod "d4392bf5-9951-4576-837d-a97043d14549" (UID: "d4392bf5-9951-4576-837d-a97043d14549"). InnerVolumeSpecName "kube-api-access-xjczd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:05.133509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.133409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4392bf5-9951-4576-837d-a97043d14549-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d4392bf5-9951-4576-837d-a97043d14549" (UID: "d4392bf5-9951-4576-837d-a97043d14549"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:05.232611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.232588    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4392bf5-9951-4576-837d-a97043d14549-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:05.232611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.232611    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjczd\" (UniqueName: \"kubernetes.io/projected/d4392bf5-9951-4576-837d-a97043d14549-kube-api-access-xjczd\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:02:05.893656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.893623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg" event={"ID":"d4392bf5-9951-4576-837d-a97043d14549","Type":"ContainerDied","Data":"883cef377e67c48e292f7de46ce1d974fe88d75479da0629078646db398b7db9"}
Sep 16 19:02:05.893656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.893640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"
Sep 16 19:02:05.894172 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.893682    2581 scope.go:117] "RemoveContainer" containerID="8aa7053d92a446c6af4cc4e93aa6465576f932b752fc623b34595722bb6312bc"
Sep 16 19:02:05.911310 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.911284    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"]
Sep 16 19:02:05.915341 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:05.915318    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6kjwg"]
Sep 16 19:02:07.221741 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:07.221705    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4392bf5-9951-4576-837d-a97043d14549" path="/var/lib/kubelet/pods/d4392bf5-9951-4576-837d-a97043d14549/volumes"
Sep 16 19:02:24.012501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012465    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"]
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012791    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012806    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012818    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerName="main"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012823    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerName="main"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012833    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="storage-initializer"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012840    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="storage-initializer"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012849    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main"
Sep 16 19:02:24.012917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012881    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main"
Sep 16 19:02:24.013177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012941    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55ad273-fd07-497a-8796-16932611dab1" containerName="main"
Sep 16 19:02:24.013177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012950    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4392bf5-9951-4576-837d-a97043d14549" containerName="main"
Sep 16 19:02:24.013177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.012957    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c30d9e9e-10ed-44bf-b442-e1c0dab64c2b" containerName="main"
Sep 16 19:02:24.017051 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.017033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.019834 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.019814    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 19:02:24.020009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.019817    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-4t4mn\""
Sep 16 19:02:24.021181 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.021162    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 19:02:24.021181 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.021171    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 19:02:24.025636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.025614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"]
Sep 16 19:02:24.072464 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.072429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7sxv\" (UniqueName: \"kubernetes.io/projected/2616fe65-c092-4e4e-b041-5d24d17970e1-kube-api-access-q7sxv\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.072635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.072478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2616fe65-c092-4e4e-b041-5d24d17970e1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.072635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.072583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.072710 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.072635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.072710 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.072661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.072775 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.072733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174361 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174361 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174361 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7sxv\" (UniqueName: \"kubernetes.io/projected/2616fe65-c092-4e4e-b041-5d24d17970e1-kube-api-access-q7sxv\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174361 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2616fe65-c092-4e4e-b041-5d24d17970e1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174708 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.174708 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.174688    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.176422 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.176399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.176665 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.176648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2616fe65-c092-4e4e-b041-5d24d17970e1-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.189328 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.189306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7sxv\" (UniqueName: \"kubernetes.io/projected/2616fe65-c092-4e4e-b041-5d24d17970e1-kube-api-access-q7sxv\") pod \"scheduler-inline-config-test-kserve-8655bc757-8ddsj\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.286057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.285972    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"]
Sep 16 19:02:24.289829 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.289806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.293215 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.293192    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-bp6pd\""
Sep 16 19:02:24.301174 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.301149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"]
Sep 16 19:02:24.327226 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.327196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:24.376541 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.376002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f5zl\" (UniqueName: \"kubernetes.io/projected/3a1f29f5-17df-43be-999f-81df9c2eed73-kube-api-access-2f5zl\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.376541 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.376069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1f29f5-17df-43be-999f-81df9c2eed73-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.477328 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.477290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f5zl\" (UniqueName: \"kubernetes.io/projected/3a1f29f5-17df-43be-999f-81df9c2eed73-kube-api-access-2f5zl\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.477496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.477399    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1f29f5-17df-43be-999f-81df9c2eed73-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.480295 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.480269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1f29f5-17df-43be-999f-81df9c2eed73-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.486034 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.486012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f5zl\" (UniqueName: \"kubernetes.io/projected/3a1f29f5-17df-43be-999f-81df9c2eed73-kube-api-access-2f5zl\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.600454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.600430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"]
Sep 16 19:02:24.600955 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.600933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:24.602821 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:02:24.602798    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2616fe65_c092_4e4e_b041_5d24d17970e1.slice/crio-031b3e0917c7a504a70c4bfe933abfb91672e8ed93ff4321822510a0014c4671 WatchSource:0}: Error finding container 031b3e0917c7a504a70c4bfe933abfb91672e8ed93ff4321822510a0014c4671: Status 404 returned error can't find the container with id 031b3e0917c7a504a70c4bfe933abfb91672e8ed93ff4321822510a0014c4671
Sep 16 19:02:24.881065 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.881041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"]
Sep 16 19:02:24.882240 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:02:24.882217    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1f29f5_17df_43be_999f_81df9c2eed73.slice/crio-820e624e925e420275111a58e1b4be80bcd3662cd5c49572374bbd54275fd89e WatchSource:0}: Error finding container 820e624e925e420275111a58e1b4be80bcd3662cd5c49572374bbd54275fd89e: Status 404 returned error can't find the container with id 820e624e925e420275111a58e1b4be80bcd3662cd5c49572374bbd54275fd89e
Sep 16 19:02:24.990469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.990434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" event={"ID":"2616fe65-c092-4e4e-b041-5d24d17970e1","Type":"ContainerStarted","Data":"4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852"}
Sep 16 19:02:24.990636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.990490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" event={"ID":"2616fe65-c092-4e4e-b041-5d24d17970e1","Type":"ContainerStarted","Data":"031b3e0917c7a504a70c4bfe933abfb91672e8ed93ff4321822510a0014c4671"}
Sep 16 19:02:24.992035 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.992001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" event={"ID":"3a1f29f5-17df-43be-999f-81df9c2eed73","Type":"ContainerStarted","Data":"5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea"}
Sep 16 19:02:24.992169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.992054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" event={"ID":"3a1f29f5-17df-43be-999f-81df9c2eed73","Type":"ContainerStarted","Data":"820e624e925e420275111a58e1b4be80bcd3662cd5c49572374bbd54275fd89e"}
Sep 16 19:02:24.992169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:24.992091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:02:25.009207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:25.009155    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" podStartSLOduration=2.009123116 podStartE2EDuration="2.009123116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:25.008547646 +0000 UTC m=+1046.312564615" watchObservedRunningTime="2026-09-16 19:02:25.009123116 +0000 UTC m=+1046.313140086"
Sep 16 19:02:25.028284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:25.028229    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podStartSLOduration=1.028210712 podStartE2EDuration="1.028210712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:25.027380313 +0000 UTC m=+1046.331397280" watchObservedRunningTime="2026-09-16 19:02:25.028210712 +0000 UTC m=+1046.332227681"
Sep 16 19:02:25.327545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:25.327511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:34.327655 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:34.327617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:34.340326 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:34.340303    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:35.052471 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:35.052443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:02:52.807835 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.807746    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"]
Sep 16 19:02:52.811438 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.811420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.814828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.814806    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 19:02:52.815058 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.815027    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-d9l85\""
Sep 16 19:02:52.815277 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.815256    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 19:02:52.815403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.815357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:02:52.827385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.827362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"]
Sep 16 19:02:52.901266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901444 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901444 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5sc\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-kube-api-access-gw5sc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901444 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901444 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901485    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901537    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:52.901639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:52.901572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002629 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002629 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002699    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.002852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.002850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5sc\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-kube-api-access-gw5sc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.003246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.003150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.003246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.003190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.003392 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.003372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.003471 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.003448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.003527 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.003460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.005015 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.004994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.005593 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.005573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.011500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.011478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.012038 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.012020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5sc\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-kube-api-access-gw5sc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-cstnp\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.123330 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.123232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:53.400933 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.400908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"]
Sep 16 19:02:53.402267 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:02:53.402242    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7920f93f_1e10_4e3f_84ac_cc3c4f8cc2cf.slice/crio-e5937a4fde6d19cedfd6d2501f3c9273f73380d995002029d67a7d7b0f542f2b WatchSource:0}: Error finding container e5937a4fde6d19cedfd6d2501f3c9273f73380d995002029d67a7d7b0f542f2b: Status 404 returned error can't find the container with id e5937a4fde6d19cedfd6d2501f3c9273f73380d995002029d67a7d7b0f542f2b
Sep 16 19:02:53.404250 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.404219    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:02:53.404344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.404287    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:02:53.404344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:53.404314    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:02:54.133791 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:54.133756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp" event={"ID":"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf","Type":"ContainerStarted","Data":"e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2"}
Sep 16 19:02:54.134228 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:54.133817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp" event={"ID":"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf","Type":"ContainerStarted","Data":"e5937a4fde6d19cedfd6d2501f3c9273f73380d995002029d67a7d7b0f542f2b"}
Sep 16 19:02:54.158262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:54.158212    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp" podStartSLOduration=2.158197561 podStartE2EDuration="2.158197561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:54.155682379 +0000 UTC m=+1075.459699372" watchObservedRunningTime="2026-09-16 19:02:54.158197561 +0000 UTC m=+1075.462214527"
Sep 16 19:02:55.123916 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:55.123875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:55.128748 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:55.128728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:55.138979 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:55.138956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:55.139631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:55.139616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:02:55.999009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:02:55.998974    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:03:16.357683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:16.357646    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"]
Sep 16 19:03:16.358135 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:16.357839    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp" podUID="7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" containerName="istio-proxy" containerID="cri-o://e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2" gracePeriod=30
Sep 16 19:03:18.109409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.109376    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:03:18.221572 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-credential-socket\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.221743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-podinfo\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.221743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw5sc\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-kube-api-access-gw5sc\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.221743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-data\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.221743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221663    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istiod-ca-cert\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.221743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-socket\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.221743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221741    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-token\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-certs\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221813    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-envoy\") pod \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\" (UID: \"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf\") "
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.221969    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.222038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.222051    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-credential-socket\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.222103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.222068    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.222453 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.222359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:18.222453 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.222411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:18.224004 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.223982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-kube-api-access-gw5sc" (OuterVolumeSpecName: "kube-api-access-gw5sc") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "kube-api-access-gw5sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:18.224104 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.224033    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:18.224148 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.224117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:03:18.224304 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.224286    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" (UID: "7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:18.259120 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.259092    2581 generic.go:356] "Generic (PLEG): container finished" podID="7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" containerID="e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2" exitCode=0
Sep 16 19:03:18.259225 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.259143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"
Sep 16 19:03:18.259225 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.259181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp" event={"ID":"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf","Type":"ContainerDied","Data":"e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2"}
Sep 16 19:03:18.259296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.259238    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp" event={"ID":"7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf","Type":"ContainerDied","Data":"e5937a4fde6d19cedfd6d2501f3c9273f73380d995002029d67a7d7b0f542f2b"}
Sep 16 19:03:18.259296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.259260    2581 scope.go:117] "RemoveContainer" containerID="e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2"
Sep 16 19:03:18.268171 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.268155    2581 scope.go:117] "RemoveContainer" containerID="e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2"
Sep 16 19:03:18.268428 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:03:18.268410    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2\": container with ID starting with e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2 not found: ID does not exist" containerID="e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2"
Sep 16 19:03:18.268470 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.268437    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2"} err="failed to get container status \"e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2\": rpc error: code = NotFound desc = could not find container \"e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2\": container with ID starting with e207739ca8e7d38b9a1da6ccba433e3cf757a58aa0bc962f59c1eef26b0d0af2 not found: ID does not exist"
Sep 16 19:03:18.281356 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.281332    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"]
Sep 16 19:03:18.285661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.285638    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-cstnp"]
Sep 16 19:03:18.322730 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322707    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-envoy\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.322730 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322727    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-podinfo\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.322913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322735    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5sc\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-kube-api-access-gw5sc\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.322913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322750    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istiod-ca-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.322913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322763    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-socket\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.322913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322772    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-istio-token\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:18.322913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:18.322781    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf-workload-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:19.222292 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:19.222256    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" path="/var/lib/kubelet/pods/7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf/volumes"
Sep 16 19:03:33.963539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:33.963507    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"]
Sep 16 19:03:33.971836 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:33.971798    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"]
Sep 16 19:03:33.973951 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:03:33.973911    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:03:33.973951 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:33.973951 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:33.973951 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852" cmd=["/bin/sleep","15"]
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:33.973976    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" message="" err=<
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:03:33.973993    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerName="main" containerID="cri-o://4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852"
Sep 16 19:03:33.974191 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:33.974039    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerName="main" containerID="cri-o://4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852" gracePeriod=60
Sep 16 19:03:34.042304 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.042257    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Sep 16 19:03:34.340158 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.340112    2581 generic.go:356] "Generic (PLEG): container finished" podID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerID="4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852" exitCode=0
Sep 16 19:03:34.340297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.340167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" event={"ID":"2616fe65-c092-4e4e-b041-5d24d17970e1","Type":"ContainerDied","Data":"4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852"}
Sep 16 19:03:34.340297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.340216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj" event={"ID":"2616fe65-c092-4e4e-b041-5d24d17970e1","Type":"ContainerDied","Data":"031b3e0917c7a504a70c4bfe933abfb91672e8ed93ff4321822510a0014c4671"}
Sep 16 19:03:34.340297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.340239    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="031b3e0917c7a504a70c4bfe933abfb91672e8ed93ff4321822510a0014c4671"
Sep 16 19:03:34.353141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.353113    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:03:34.456142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-dshm\") pod \"2616fe65-c092-4e4e-b041-5d24d17970e1\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") "
Sep 16 19:03:34.456332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-home\") pod \"2616fe65-c092-4e4e-b041-5d24d17970e1\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") "
Sep 16 19:03:34.456332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456201    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-tmp-dir\") pod \"2616fe65-c092-4e4e-b041-5d24d17970e1\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") "
Sep 16 19:03:34.456332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456224    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2616fe65-c092-4e4e-b041-5d24d17970e1-tls-certs\") pod \"2616fe65-c092-4e4e-b041-5d24d17970e1\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") "
Sep 16 19:03:34.456332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7sxv\" (UniqueName: \"kubernetes.io/projected/2616fe65-c092-4e4e-b041-5d24d17970e1-kube-api-access-q7sxv\") pod \"2616fe65-c092-4e4e-b041-5d24d17970e1\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") "
Sep 16 19:03:34.456332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-model-cache\") pod \"2616fe65-c092-4e4e-b041-5d24d17970e1\" (UID: \"2616fe65-c092-4e4e-b041-5d24d17970e1\") "
Sep 16 19:03:34.456591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456507    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-home" (OuterVolumeSpecName: "home") pod "2616fe65-c092-4e4e-b041-5d24d17970e1" (UID: "2616fe65-c092-4e4e-b041-5d24d17970e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:34.456591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2616fe65-c092-4e4e-b041-5d24d17970e1" (UID: "2616fe65-c092-4e4e-b041-5d24d17970e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:34.456704 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.456618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "2616fe65-c092-4e4e-b041-5d24d17970e1" (UID: "2616fe65-c092-4e4e-b041-5d24d17970e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:34.458625 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.458578    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2616fe65-c092-4e4e-b041-5d24d17970e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2616fe65-c092-4e4e-b041-5d24d17970e1" (UID: "2616fe65-c092-4e4e-b041-5d24d17970e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:34.458747 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.458674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-dshm" (OuterVolumeSpecName: "dshm") pod "2616fe65-c092-4e4e-b041-5d24d17970e1" (UID: "2616fe65-c092-4e4e-b041-5d24d17970e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:34.458747 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.458675    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2616fe65-c092-4e4e-b041-5d24d17970e1-kube-api-access-q7sxv" (OuterVolumeSpecName: "kube-api-access-q7sxv") pod "2616fe65-c092-4e4e-b041-5d24d17970e1" (UID: "2616fe65-c092-4e4e-b041-5d24d17970e1"). InnerVolumeSpecName "kube-api-access-q7sxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:34.557750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.557666    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:34.557750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.557696    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:34.557750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.557706    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2616fe65-c092-4e4e-b041-5d24d17970e1-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:34.557750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.557715    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7sxv\" (UniqueName: \"kubernetes.io/projected/2616fe65-c092-4e4e-b041-5d24d17970e1-kube-api-access-q7sxv\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:34.557750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.557723    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:34.557750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:34.557731    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2616fe65-c092-4e4e-b041-5d24d17970e1-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:03:35.345603 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:35.345568    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"
Sep 16 19:03:35.362994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:35.362958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"]
Sep 16 19:03:35.366147 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:35.366121    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8ddsj"]
Sep 16 19:03:35.998786 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:35.998747    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:37.221825 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:37.221789    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" path="/var/lib/kubelet/pods/2616fe65-c092-4e4e-b041-5d24d17970e1/volumes"
Sep 16 19:03:45.999225 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:45.999182    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:48.991998 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:48.991939    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main" containerID="cri-o://5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea" gracePeriod=45
Sep 16 19:03:55.998599 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:55.998553    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:55.999040 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:03:55.998675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:04:05.998275 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:05.998234    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:15.999169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:15.999133    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:19.258435 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.258369    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:04:19.408017 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.407978    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f5zl\" (UniqueName: \"kubernetes.io/projected/3a1f29f5-17df-43be-999f-81df9c2eed73-kube-api-access-2f5zl\") pod \"3a1f29f5-17df-43be-999f-81df9c2eed73\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") "
Sep 16 19:04:19.408214 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.408045    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1f29f5-17df-43be-999f-81df9c2eed73-tls-certs\") pod \"3a1f29f5-17df-43be-999f-81df9c2eed73\" (UID: \"3a1f29f5-17df-43be-999f-81df9c2eed73\") "
Sep 16 19:04:19.410132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.410105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1f29f5-17df-43be-999f-81df9c2eed73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a1f29f5-17df-43be-999f-81df9c2eed73" (UID: "3a1f29f5-17df-43be-999f-81df9c2eed73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:19.410217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.410151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1f29f5-17df-43be-999f-81df9c2eed73-kube-api-access-2f5zl" (OuterVolumeSpecName: "kube-api-access-2f5zl") pod "3a1f29f5-17df-43be-999f-81df9c2eed73" (UID: "3a1f29f5-17df-43be-999f-81df9c2eed73"). InnerVolumeSpecName "kube-api-access-2f5zl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:19.508557 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.508453    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f5zl\" (UniqueName: \"kubernetes.io/projected/3a1f29f5-17df-43be-999f-81df9c2eed73-kube-api-access-2f5zl\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:19.508557 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.508497    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a1f29f5-17df-43be-999f-81df9c2eed73-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:04:19.565101 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.565068    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerID="5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea" exitCode=0
Sep 16 19:04:19.565290 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.565112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"
Sep 16 19:04:19.565290 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.565151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" event={"ID":"3a1f29f5-17df-43be-999f-81df9c2eed73","Type":"ContainerDied","Data":"5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea"}
Sep 16 19:04:19.565290 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.565195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh" event={"ID":"3a1f29f5-17df-43be-999f-81df9c2eed73","Type":"ContainerDied","Data":"820e624e925e420275111a58e1b4be80bcd3662cd5c49572374bbd54275fd89e"}
Sep 16 19:04:19.565290 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.565218    2581 scope.go:117] "RemoveContainer" containerID="5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea"
Sep 16 19:04:19.574150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.574134    2581 scope.go:117] "RemoveContainer" containerID="5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea"
Sep 16 19:04:19.574426 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:04:19.574405    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea\": container with ID starting with 5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea not found: ID does not exist" containerID="5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea"
Sep 16 19:04:19.574479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.574433    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea"} err="failed to get container status \"5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea\": rpc error: code = NotFound desc = could not find container \"5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea\": container with ID starting with 5152ec5ed46bbe3550faab23ec5c53d5fcb123a092c96b7e120b3eabf1db84ea not found: ID does not exist"
Sep 16 19:04:19.586575 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.586545    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"]
Sep 16 19:04:19.591048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:19.591020    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbcg5bh"]
Sep 16 19:04:21.221896 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:21.221840    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" path="/var/lib/kubelet/pods/3a1f29f5-17df-43be-999f-81df9c2eed73/volumes"
Sep 16 19:04:25.176045 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176007    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"]
Sep 16 19:04:25.176415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerName="main"
Sep 16 19:04:25.176415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176337    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerName="main"
Sep 16 19:04:25.176415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176357    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" containerName="istio-proxy"
Sep 16 19:04:25.176415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176362    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" containerName="istio-proxy"
Sep 16 19:04:25.176415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main"
Sep 16 19:04:25.176415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176376    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main"
Sep 16 19:04:25.176604 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176421    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2616fe65-c092-4e4e-b041-5d24d17970e1" containerName="main"
Sep 16 19:04:25.176604 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176428    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a1f29f5-17df-43be-999f-81df9c2eed73" containerName="main"
Sep 16 19:04:25.176604 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.176435    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7920f93f-1e10-4e3f-84ac-cc3c4f8cc2cf" containerName="istio-proxy"
Sep 16 19:04:25.180751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.180733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.183886 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.183839    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:04:25.184975 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.184949    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:04:25.185107 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.184955    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-qht7k\""
Sep 16 19:04:25.185107 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.185013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:04:25.191332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.191307    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"]
Sep 16 19:04:25.246724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.246691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.246921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.246738    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.246921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.246792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.246921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.246832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.246921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.246853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12829ee-f260-43b3-a05c-3a45f6818956-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.246921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.246905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7p2x\" (UniqueName: \"kubernetes.io/projected/f12829ee-f260-43b3-a05c-3a45f6818956-kube-api-access-z7p2x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.347789 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.347748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348013 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.347805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12829ee-f260-43b3-a05c-3a45f6818956-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348013 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.347881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7p2x\" (UniqueName: \"kubernetes.io/projected/f12829ee-f260-43b3-a05c-3a45f6818956-kube-api-access-z7p2x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348013 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.347927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348013 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.347977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348238 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.348021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348379 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.348351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348519 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.348385    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.348519 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.348418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.350066 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.350045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.350369 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.350348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12829ee-f260-43b3-a05c-3a45f6818956-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.360978 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.360953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7p2x\" (UniqueName: \"kubernetes.io/projected/f12829ee-f260-43b3-a05c-3a45f6818956-kube-api-access-z7p2x\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.381286 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.381258    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"]
Sep 16 19:04:25.384746 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.384728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.387932 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.387899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-wzvqk\""
Sep 16 19:04:25.396557 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.396537    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"]
Sep 16 19:04:25.448461 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.448380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlpd\" (UniqueName: \"kubernetes.io/projected/c90238e0-182b-4781-8d06-1c08cc2f6e06-kube-api-access-rmlpd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.448461 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.448444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c90238e0-182b-4781-8d06-1c08cc2f6e06-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.491460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.491418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:25.549625 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.549593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c90238e0-182b-4781-8d06-1c08cc2f6e06-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.549737 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.549645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlpd\" (UniqueName: \"kubernetes.io/projected/c90238e0-182b-4781-8d06-1c08cc2f6e06-kube-api-access-rmlpd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.555195 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.555168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c90238e0-182b-4781-8d06-1c08cc2f6e06-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.559774 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.559707    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlpd\" (UniqueName: \"kubernetes.io/projected/c90238e0-182b-4781-8d06-1c08cc2f6e06-kube-api-access-rmlpd\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.697087 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.697050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:25.772639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.772614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"]
Sep 16 19:04:25.774795 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:04:25.774758    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12829ee_f260_43b3_a05c_3a45f6818956.slice/crio-008c3b161827e27443c1daa4824d3653ac0083690a56db82639fd6f01f018c19 WatchSource:0}: Error finding container 008c3b161827e27443c1daa4824d3653ac0083690a56db82639fd6f01f018c19: Status 404 returned error can't find the container with id 008c3b161827e27443c1daa4824d3653ac0083690a56db82639fd6f01f018c19
Sep 16 19:04:25.970899 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:25.970868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"]
Sep 16 19:04:25.972588 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:04:25.972561    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90238e0_182b_4781_8d06_1c08cc2f6e06.slice/crio-5d5d2b26010cd2ba453b419c3c9586895d37ccc7f97c79dbe4c919739d34c596 WatchSource:0}: Error finding container 5d5d2b26010cd2ba453b419c3c9586895d37ccc7f97c79dbe4c919739d34c596: Status 404 returned error can't find the container with id 5d5d2b26010cd2ba453b419c3c9586895d37ccc7f97c79dbe4c919739d34c596
Sep 16 19:04:26.604057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.604020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" event={"ID":"c90238e0-182b-4781-8d06-1c08cc2f6e06","Type":"ContainerStarted","Data":"163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329"}
Sep 16 19:04:26.604527 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.604088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:26.604527 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.604109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" event={"ID":"c90238e0-182b-4781-8d06-1c08cc2f6e06","Type":"ContainerStarted","Data":"5d5d2b26010cd2ba453b419c3c9586895d37ccc7f97c79dbe4c919739d34c596"}
Sep 16 19:04:26.605685 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.605660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" event={"ID":"f12829ee-f260-43b3-a05c-3a45f6818956","Type":"ContainerStarted","Data":"20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6"}
Sep 16 19:04:26.605806 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.605705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" event={"ID":"f12829ee-f260-43b3-a05c-3a45f6818956","Type":"ContainerStarted","Data":"008c3b161827e27443c1daa4824d3653ac0083690a56db82639fd6f01f018c19"}
Sep 16 19:04:26.623357 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.623311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" podStartSLOduration=1.623298334 podStartE2EDuration="1.623298334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:26.620702002 +0000 UTC m=+1167.924719006" watchObservedRunningTime="2026-09-16 19:04:26.623298334 +0000 UTC m=+1167.927315300"
Sep 16 19:04:26.639147 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:26.639109    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" podStartSLOduration=1.639095202 podStartE2EDuration="1.639095202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:26.637315001 +0000 UTC m=+1167.941331993" watchObservedRunningTime="2026-09-16 19:04:26.639095202 +0000 UTC m=+1167.943112169"
Sep 16 19:04:27.491993 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:27.491937    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:35.491743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:35.491708    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:35.504183 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:35.504155    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:35.672530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:35.672500    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:04:57.622751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:57.622721    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:04:59.229250 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.229221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:04:59.232264 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.232241    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:04:59.232760 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.232740    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:04:59.235299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.235271    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:04:59.298529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.298498    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"]
Sep 16 19:04:59.303298 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.303269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"]
Sep 16 19:04:59.311210 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:04:59.311173    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:04:59.311210 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:59.311210 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:59.311210 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6" cmd=["/bin/sleep","15"]
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.311228    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" message="" err=<
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:04:59.311245    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" containerName="main" containerID="cri-o://20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6"
Sep 16 19:04:59.311431 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.311285    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" containerName="main" containerID="cri-o://20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6" gracePeriod=60
Sep 16 19:04:59.662334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.662298    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/health\": dial tcp 10.132.0.43:8000: connect: connection refused"
Sep 16 19:04:59.782566 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.782535    2581 generic.go:356] "Generic (PLEG): container finished" podID="f12829ee-f260-43b3-a05c-3a45f6818956" containerID="20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6" exitCode=0
Sep 16 19:04:59.782787 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:04:59.782613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" event={"ID":"f12829ee-f260-43b3-a05c-3a45f6818956","Type":"ContainerDied","Data":"20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6"}
Sep 16 19:05:00.047596 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.047568    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:05:00.105075 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-model-cache\") pod \"f12829ee-f260-43b3-a05c-3a45f6818956\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") "
Sep 16 19:05:00.105263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-home\") pod \"f12829ee-f260-43b3-a05c-3a45f6818956\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") "
Sep 16 19:05:00.105263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105114    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-tmp-dir\") pod \"f12829ee-f260-43b3-a05c-3a45f6818956\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") "
Sep 16 19:05:00.105263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-dshm\") pod \"f12829ee-f260-43b3-a05c-3a45f6818956\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") "
Sep 16 19:05:00.105263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105180    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12829ee-f260-43b3-a05c-3a45f6818956-tls-certs\") pod \"f12829ee-f260-43b3-a05c-3a45f6818956\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") "
Sep 16 19:05:00.105263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105205    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7p2x\" (UniqueName: \"kubernetes.io/projected/f12829ee-f260-43b3-a05c-3a45f6818956-kube-api-access-z7p2x\") pod \"f12829ee-f260-43b3-a05c-3a45f6818956\" (UID: \"f12829ee-f260-43b3-a05c-3a45f6818956\") "
Sep 16 19:05:00.105500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105359    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-model-cache" (OuterVolumeSpecName: "model-cache") pod "f12829ee-f260-43b3-a05c-3a45f6818956" (UID: "f12829ee-f260-43b3-a05c-3a45f6818956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:00.105500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-home" (OuterVolumeSpecName: "home") pod "f12829ee-f260-43b3-a05c-3a45f6818956" (UID: "f12829ee-f260-43b3-a05c-3a45f6818956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:00.105500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.105451    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f12829ee-f260-43b3-a05c-3a45f6818956" (UID: "f12829ee-f260-43b3-a05c-3a45f6818956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:00.107316 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.107288    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-dshm" (OuterVolumeSpecName: "dshm") pod "f12829ee-f260-43b3-a05c-3a45f6818956" (UID: "f12829ee-f260-43b3-a05c-3a45f6818956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:05:00.107419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.107381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12829ee-f260-43b3-a05c-3a45f6818956-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f12829ee-f260-43b3-a05c-3a45f6818956" (UID: "f12829ee-f260-43b3-a05c-3a45f6818956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:00.107419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.107390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12829ee-f260-43b3-a05c-3a45f6818956-kube-api-access-z7p2x" (OuterVolumeSpecName: "kube-api-access-z7p2x") pod "f12829ee-f260-43b3-a05c-3a45f6818956" (UID: "f12829ee-f260-43b3-a05c-3a45f6818956"). InnerVolumeSpecName "kube-api-access-z7p2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:00.205812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.205746    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:00.205812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.205768    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f12829ee-f260-43b3-a05c-3a45f6818956-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:00.205812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.205778    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7p2x\" (UniqueName: \"kubernetes.io/projected/f12829ee-f260-43b3-a05c-3a45f6818956-kube-api-access-z7p2x\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:00.205812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.205787    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:00.205812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.205795    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:00.206127 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.205821    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f12829ee-f260-43b3-a05c-3a45f6818956-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:00.789028 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.788994    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"
Sep 16 19:05:00.789498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.789031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm" event={"ID":"f12829ee-f260-43b3-a05c-3a45f6818956","Type":"ContainerDied","Data":"008c3b161827e27443c1daa4824d3653ac0083690a56db82639fd6f01f018c19"}
Sep 16 19:05:00.789498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.789095    2581 scope.go:117] "RemoveContainer" containerID="20b70a9ffd86a347bc1dfc5b2293bb8b65cdfb6c066b726fe1254ba0bb5d1fd6"
Sep 16 19:05:00.810561 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.810537    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"]
Sep 16 19:05:00.814534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:00.814510    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fpr4dm"]
Sep 16 19:05:01.226792 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:01.223391    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" path="/var/lib/kubelet/pods/f12829ee-f260-43b3-a05c-3a45f6818956/volumes"
Sep 16 19:05:07.623177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:07.623136    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:14.326426 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:14.326370    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main" containerID="cri-o://163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329" gracePeriod=45
Sep 16 19:05:17.623236 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:17.623197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:27.622682 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:27.622638    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:27.623109 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:27.622751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:05:37.623547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:37.623508    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:05:44.575599 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.575573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:05:44.731239 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.731211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmlpd\" (UniqueName: \"kubernetes.io/projected/c90238e0-182b-4781-8d06-1c08cc2f6e06-kube-api-access-rmlpd\") pod \"c90238e0-182b-4781-8d06-1c08cc2f6e06\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") "
Sep 16 19:05:44.731387 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.731263    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c90238e0-182b-4781-8d06-1c08cc2f6e06-tls-certs\") pod \"c90238e0-182b-4781-8d06-1c08cc2f6e06\" (UID: \"c90238e0-182b-4781-8d06-1c08cc2f6e06\") "
Sep 16 19:05:44.733240 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.733214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90238e0-182b-4781-8d06-1c08cc2f6e06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c90238e0-182b-4781-8d06-1c08cc2f6e06" (UID: "c90238e0-182b-4781-8d06-1c08cc2f6e06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:05:44.733343 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.733241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90238e0-182b-4781-8d06-1c08cc2f6e06-kube-api-access-rmlpd" (OuterVolumeSpecName: "kube-api-access-rmlpd") pod "c90238e0-182b-4781-8d06-1c08cc2f6e06" (UID: "c90238e0-182b-4781-8d06-1c08cc2f6e06"). InnerVolumeSpecName "kube-api-access-rmlpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:05:44.832246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.832223    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmlpd\" (UniqueName: \"kubernetes.io/projected/c90238e0-182b-4781-8d06-1c08cc2f6e06-kube-api-access-rmlpd\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:44.832246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:44.832244    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c90238e0-182b-4781-8d06-1c08cc2f6e06-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:05:45.005037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.004952    2581 generic.go:356] "Generic (PLEG): container finished" podID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerID="163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329" exitCode=0
Sep 16 19:05:45.005037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.005008    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"
Sep 16 19:05:45.005205 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.005036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" event={"ID":"c90238e0-182b-4781-8d06-1c08cc2f6e06","Type":"ContainerDied","Data":"163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329"}
Sep 16 19:05:45.005205 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.005092    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx" event={"ID":"c90238e0-182b-4781-8d06-1c08cc2f6e06","Type":"ContainerDied","Data":"5d5d2b26010cd2ba453b419c3c9586895d37ccc7f97c79dbe4c919739d34c596"}
Sep 16 19:05:45.005205 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.005122    2581 scope.go:117] "RemoveContainer" containerID="163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329"
Sep 16 19:05:45.013608 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.013590    2581 scope.go:117] "RemoveContainer" containerID="163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329"
Sep 16 19:05:45.013892 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:05:45.013848    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329\": container with ID starting with 163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329 not found: ID does not exist" containerID="163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329"
Sep 16 19:05:45.013969 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.013903    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329"} err="failed to get container status \"163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329\": rpc error: code = NotFound desc = could not find container \"163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329\": container with ID starting with 163801abf047217e939e28bc1178da7d51c69124fb317b4a4d7794027d737329 not found: ID does not exist"
Sep 16 19:05:45.026256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.026232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"]
Sep 16 19:05:45.030150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.030128    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schez9wwx"]
Sep 16 19:05:45.221844 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:45.221809    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" path="/var/lib/kubelet/pods/c90238e0-182b-4781-8d06-1c08cc2f6e06/volumes"
Sep 16 19:05:57.611770 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.611696    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"]
Sep 16 19:05:57.612146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.612057    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" containerName="main"
Sep 16 19:05:57.612146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.612070    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" containerName="main"
Sep 16 19:05:57.612146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.612078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main"
Sep 16 19:05:57.612146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.612084    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main"
Sep 16 19:05:57.612146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.612135    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90238e0-182b-4781-8d06-1c08cc2f6e06" containerName="main"
Sep 16 19:05:57.612146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.612145    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12829ee-f260-43b3-a05c-3a45f6818956" containerName="main"
Sep 16 19:05:57.616307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.616290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.619208 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.619187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:05:57.620305 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.620288    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:05:57.620414 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.620373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-nm8lx\""
Sep 16 19:05:57.620414 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.620386    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:05:57.627185 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.627164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"]
Sep 16 19:05:57.734354 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.734329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.734501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.734357    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.734501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.734380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534afe81-0b0a-402a-b606-2fb56541268b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.734501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.734399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.734501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.734488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.734672 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.734536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kv8\" (UniqueName: \"kubernetes.io/projected/534afe81-0b0a-402a-b606-2fb56541268b-kube-api-access-85kv8\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.835498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.835463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85kv8\" (UniqueName: \"kubernetes.io/projected/534afe81-0b0a-402a-b606-2fb56541268b-kube-api-access-85kv8\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.835644 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.835523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.835644 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.835623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.835779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.835683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534afe81-0b0a-402a-b606-2fb56541268b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.835779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.835716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.835906 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.835778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.836085 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.836065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.836170 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.836128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.836224 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.836171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.837595 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.837577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.838110 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.838092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534afe81-0b0a-402a-b606-2fb56541268b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.844176 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.844151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kv8\" (UniqueName: \"kubernetes.io/projected/534afe81-0b0a-402a-b606-2fb56541268b-kube-api-access-85kv8\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:57.870539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.870476    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"]
Sep 16 19:05:57.874292 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.874274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:57.877445 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.877424    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-tszdf\""
Sep 16 19:05:57.887593 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.887571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"]
Sep 16 19:05:57.927501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:57.927475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:05:58.037258 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.037227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbx96\" (UniqueName: \"kubernetes.io/projected/313016a2-8593-4744-b55e-7134ed94c8fd-kube-api-access-tbx96\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.037415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.037279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313016a2-8593-4744-b55e-7134ed94c8fd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.137848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.137755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbx96\" (UniqueName: \"kubernetes.io/projected/313016a2-8593-4744-b55e-7134ed94c8fd-kube-api-access-tbx96\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.137848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.137839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313016a2-8593-4744-b55e-7134ed94c8fd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.140219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.140192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313016a2-8593-4744-b55e-7134ed94c8fd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.146119 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.146091    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbx96\" (UniqueName: \"kubernetes.io/projected/313016a2-8593-4744-b55e-7134ed94c8fd-kube-api-access-tbx96\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.184820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.184791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:58.200069 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.200041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"]
Sep 16 19:05:58.204296 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:05:58.204266    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534afe81_0b0a_402a_b606_2fb56541268b.slice/crio-1cb5b67adc8e6ca46933b8478dbb64f588d9516875fa95504b4dca6274628229 WatchSource:0}: Error finding container 1cb5b67adc8e6ca46933b8478dbb64f588d9516875fa95504b4dca6274628229: Status 404 returned error can't find the container with id 1cb5b67adc8e6ca46933b8478dbb64f588d9516875fa95504b4dca6274628229
Sep 16 19:05:58.468275 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:58.468249    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"]
Sep 16 19:05:58.470023 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:05:58.469999    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313016a2_8593_4744_b55e_7134ed94c8fd.slice/crio-aefdb16f8a0ffb2ab38346ba1ada0b99643fbace13cf239b6ff986e27d174b58 WatchSource:0}: Error finding container aefdb16f8a0ffb2ab38346ba1ada0b99643fbace13cf239b6ff986e27d174b58: Status 404 returned error can't find the container with id aefdb16f8a0ffb2ab38346ba1ada0b99643fbace13cf239b6ff986e27d174b58
Sep 16 19:05:59.079190 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.079150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" event={"ID":"313016a2-8593-4744-b55e-7134ed94c8fd","Type":"ContainerStarted","Data":"ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152"}
Sep 16 19:05:59.079618 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.079203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" event={"ID":"313016a2-8593-4744-b55e-7134ed94c8fd","Type":"ContainerStarted","Data":"aefdb16f8a0ffb2ab38346ba1ada0b99643fbace13cf239b6ff986e27d174b58"}
Sep 16 19:05:59.079618 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.079226    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:05:59.080699 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.080675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" event={"ID":"534afe81-0b0a-402a-b606-2fb56541268b","Type":"ContainerStarted","Data":"53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"}
Sep 16 19:05:59.080828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.080718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" event={"ID":"534afe81-0b0a-402a-b606-2fb56541268b","Type":"ContainerStarted","Data":"1cb5b67adc8e6ca46933b8478dbb64f588d9516875fa95504b4dca6274628229"}
Sep 16 19:05:59.097912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.097870    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" podStartSLOduration=2.097842915 podStartE2EDuration="2.097842915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:59.096369968 +0000 UTC m=+1260.400386975" watchObservedRunningTime="2026-09-16 19:05:59.097842915 +0000 UTC m=+1260.401859876"
Sep 16 19:05:59.113282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.113241    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" podStartSLOduration=2.113228188 podStartE2EDuration="2.113228188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:59.112991781 +0000 UTC m=+1260.417008750" watchObservedRunningTime="2026-09-16 19:05:59.113228188 +0000 UTC m=+1260.417245154"
Sep 16 19:05:59.928524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:05:59.928492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:06:07.928180 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:07.928144    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:06:07.940611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:07.940590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:06:08.138466 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:08.138429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:06:30.089609 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:30.089570    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:06:31.652506 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:31.652462    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"]
Sep 16 19:06:31.655497 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:31.655466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"]
Sep 16 19:06:31.662282 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:06:31.662236    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:06:31.662282 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:31.662282 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:31.662282 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418" cmd=["/bin/sleep","15"]
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:31.662304    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" message="" err=<
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:06:31.662323    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" podUID="534afe81-0b0a-402a-b606-2fb56541268b" containerName="main" containerID="cri-o://53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"
Sep 16 19:06:31.662569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:31.662367    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" podUID="534afe81-0b0a-402a-b606-2fb56541268b" containerName="main" containerID="cri-o://53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418" gracePeriod=60
Sep 16 19:06:32.009140 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.009111    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:06:32.096409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-model-cache\") pod \"534afe81-0b0a-402a-b606-2fb56541268b\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") "
Sep 16 19:06:32.096589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-dshm\") pod \"534afe81-0b0a-402a-b606-2fb56541268b\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") "
Sep 16 19:06:32.096589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096472    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-home\") pod \"534afe81-0b0a-402a-b606-2fb56541268b\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") "
Sep 16 19:06:32.096589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-tmp-dir\") pod \"534afe81-0b0a-402a-b606-2fb56541268b\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") "
Sep 16 19:06:32.096589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85kv8\" (UniqueName: \"kubernetes.io/projected/534afe81-0b0a-402a-b606-2fb56541268b-kube-api-access-85kv8\") pod \"534afe81-0b0a-402a-b606-2fb56541268b\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") "
Sep 16 19:06:32.096589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534afe81-0b0a-402a-b606-2fb56541268b-tls-certs\") pod \"534afe81-0b0a-402a-b606-2fb56541268b\" (UID: \"534afe81-0b0a-402a-b606-2fb56541268b\") "
Sep 16 19:06:32.096878 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096658    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-model-cache" (OuterVolumeSpecName: "model-cache") pod "534afe81-0b0a-402a-b606-2fb56541268b" (UID: "534afe81-0b0a-402a-b606-2fb56541268b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:32.096878 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096785    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:06:32.096878 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096785    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-home" (OuterVolumeSpecName: "home") pod "534afe81-0b0a-402a-b606-2fb56541268b" (UID: "534afe81-0b0a-402a-b606-2fb56541268b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:32.096878 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.096800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "534afe81-0b0a-402a-b606-2fb56541268b" (UID: "534afe81-0b0a-402a-b606-2fb56541268b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:32.098794 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.098772    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-dshm" (OuterVolumeSpecName: "dshm") pod "534afe81-0b0a-402a-b606-2fb56541268b" (UID: "534afe81-0b0a-402a-b606-2fb56541268b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:32.098794 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.098785    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534afe81-0b0a-402a-b606-2fb56541268b-kube-api-access-85kv8" (OuterVolumeSpecName: "kube-api-access-85kv8") pod "534afe81-0b0a-402a-b606-2fb56541268b" (UID: "534afe81-0b0a-402a-b606-2fb56541268b"). InnerVolumeSpecName "kube-api-access-85kv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:32.098936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.098819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534afe81-0b0a-402a-b606-2fb56541268b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "534afe81-0b0a-402a-b606-2fb56541268b" (UID: "534afe81-0b0a-402a-b606-2fb56541268b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:32.198142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.198057    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85kv8\" (UniqueName: \"kubernetes.io/projected/534afe81-0b0a-402a-b606-2fb56541268b-kube-api-access-85kv8\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:06:32.198142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.198087    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/534afe81-0b0a-402a-b606-2fb56541268b-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:06:32.198142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.198098    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:06:32.198142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.198106    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:06:32.198142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.198135    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/534afe81-0b0a-402a-b606-2fb56541268b-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:06:32.247829 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.247791    2581 generic.go:356] "Generic (PLEG): container finished" podID="534afe81-0b0a-402a-b606-2fb56541268b" containerID="53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418" exitCode=0
Sep 16 19:06:32.248032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.247837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" event={"ID":"534afe81-0b0a-402a-b606-2fb56541268b","Type":"ContainerDied","Data":"53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"}
Sep 16 19:06:32.248032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.247847    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"
Sep 16 19:06:32.248032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.247903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt" event={"ID":"534afe81-0b0a-402a-b606-2fb56541268b","Type":"ContainerDied","Data":"1cb5b67adc8e6ca46933b8478dbb64f588d9516875fa95504b4dca6274628229"}
Sep 16 19:06:32.248032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.247932    2581 scope.go:117] "RemoveContainer" containerID="53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"
Sep 16 19:06:32.259762 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.259746    2581 scope.go:117] "RemoveContainer" containerID="53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"
Sep 16 19:06:32.260087 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:06:32.260067    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418\": container with ID starting with 53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418 not found: ID does not exist" containerID="53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"
Sep 16 19:06:32.260144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.260096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418"} err="failed to get container status \"53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418\": rpc error: code = NotFound desc = could not find container \"53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418\": container with ID starting with 53b683ec23c6d79148c65be0c9737f793e8d2cf60cbf209f7bc9e42e260fe418 not found: ID does not exist"
Sep 16 19:06:32.270255 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.270228    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"]
Sep 16 19:06:32.272410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:32.272388    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-s2vgt"]
Sep 16 19:06:33.221406 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:33.221372    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="534afe81-0b0a-402a-b606-2fb56541268b" path="/var/lib/kubelet/pods/534afe81-0b0a-402a-b606-2fb56541268b/volumes"
Sep 16 19:06:40.090005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:40.089965    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:46.674618 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:46.674580    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main" containerID="cri-o://ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152" gracePeriod=45
Sep 16 19:06:50.089980 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:06:50.089941    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:00.090160 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:00.090119    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:00.090525 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:00.090258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:07:10.089910 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:10.089847    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:07:16.914649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:16.914624    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:07:17.060024 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.059935    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbx96\" (UniqueName: \"kubernetes.io/projected/313016a2-8593-4744-b55e-7134ed94c8fd-kube-api-access-tbx96\") pod \"313016a2-8593-4744-b55e-7134ed94c8fd\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") "
Sep 16 19:07:17.060202 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.060030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313016a2-8593-4744-b55e-7134ed94c8fd-tls-certs\") pod \"313016a2-8593-4744-b55e-7134ed94c8fd\" (UID: \"313016a2-8593-4744-b55e-7134ed94c8fd\") "
Sep 16 19:07:17.062028 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.061994    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/313016a2-8593-4744-b55e-7134ed94c8fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "313016a2-8593-4744-b55e-7134ed94c8fd" (UID: "313016a2-8593-4744-b55e-7134ed94c8fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:07:17.062028 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.062013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313016a2-8593-4744-b55e-7134ed94c8fd-kube-api-access-tbx96" (OuterVolumeSpecName: "kube-api-access-tbx96") pod "313016a2-8593-4744-b55e-7134ed94c8fd" (UID: "313016a2-8593-4744-b55e-7134ed94c8fd"). InnerVolumeSpecName "kube-api-access-tbx96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:07:17.161534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.161500    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbx96\" (UniqueName: \"kubernetes.io/projected/313016a2-8593-4744-b55e-7134ed94c8fd-kube-api-access-tbx96\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:07:17.161534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.161529    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313016a2-8593-4744-b55e-7134ed94c8fd-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:07:17.491240 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.491204    2581 generic.go:356] "Generic (PLEG): container finished" podID="313016a2-8593-4744-b55e-7134ed94c8fd" containerID="ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152" exitCode=0
Sep 16 19:07:17.491420 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.491251    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"
Sep 16 19:07:17.491420 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.491285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" event={"ID":"313016a2-8593-4744-b55e-7134ed94c8fd","Type":"ContainerDied","Data":"ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152"}
Sep 16 19:07:17.491420 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.491340    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz" event={"ID":"313016a2-8593-4744-b55e-7134ed94c8fd","Type":"ContainerDied","Data":"aefdb16f8a0ffb2ab38346ba1ada0b99643fbace13cf239b6ff986e27d174b58"}
Sep 16 19:07:17.491420 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.491362    2581 scope.go:117] "RemoveContainer" containerID="ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152"
Sep 16 19:07:17.501611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.501573    2581 scope.go:117] "RemoveContainer" containerID="ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152"
Sep 16 19:07:17.502064 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:07:17.502037    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152\": container with ID starting with ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152 not found: ID does not exist" containerID="ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152"
Sep 16 19:07:17.502174 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.502064    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152"} err="failed to get container status \"ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152\": rpc error: code = NotFound desc = could not find container \"ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152\": container with ID starting with ccc4670406c76704918c992fe0a0da99c6d20c8a4a6ecea368896e94cb854152 not found: ID does not exist"
Sep 16 19:07:17.508160 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.508138    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"]
Sep 16 19:07:17.514069 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:17.514048    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9sq6gz"]
Sep 16 19:07:19.223077 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:19.223044    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" path="/var/lib/kubelet/pods/313016a2-8593-4744-b55e-7134ed94c8fd/volumes"
Sep 16 19:07:30.804794 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.804704    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"]
Sep 16 19:07:30.805254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.805051    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main"
Sep 16 19:07:30.805254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.805063    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main"
Sep 16 19:07:30.805254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.805077    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="534afe81-0b0a-402a-b606-2fb56541268b" containerName="main"
Sep 16 19:07:30.805254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.805082    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="534afe81-0b0a-402a-b606-2fb56541268b" containerName="main"
Sep 16 19:07:30.805254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.805130    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="534afe81-0b0a-402a-b606-2fb56541268b" containerName="main"
Sep 16 19:07:30.805254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.805138    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="313016a2-8593-4744-b55e-7134ed94c8fd" containerName="main"
Sep 16 19:07:30.808023 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.808000    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.811008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.810984    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:07:30.811167 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.811081    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:07:30.812339 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.812320    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-nhj5b\""
Sep 16 19:07:30.812339 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.812327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:07:30.820519 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.820497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"]
Sep 16 19:07:30.865300 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.865272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.865425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.865307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.865425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.865344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.865425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.865361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fct6\" (UniqueName: \"kubernetes.io/projected/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-kube-api-access-4fct6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.865425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.865397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.865425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.865416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.965815 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.965777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.965815 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.965814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fct6\" (UniqueName: \"kubernetes.io/projected/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-kube-api-access-4fct6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.965837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.965881    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.965924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.965944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.966266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.966310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.966414 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.966328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.968206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.968186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.968558 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.968535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:30.974798 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:30.974773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fct6\" (UniqueName: \"kubernetes.io/projected/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-kube-api-access-4fct6\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:31.031284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.031250    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"]
Sep 16 19:07:31.034698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.034677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.037450 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.037419    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hpkpc\""
Sep 16 19:07:31.044143 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.044119    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"]
Sep 16 19:07:31.067000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.066932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963bd392-2b78-4fda-8c15-322725ef0e66-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.067000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.066971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plw2s\" (UniqueName: \"kubernetes.io/projected/963bd392-2b78-4fda-8c15-322725ef0e66-kube-api-access-plw2s\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.118751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.118720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:31.168647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.167832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963bd392-2b78-4fda-8c15-322725ef0e66-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.168647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.167909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plw2s\" (UniqueName: \"kubernetes.io/projected/963bd392-2b78-4fda-8c15-322725ef0e66-kube-api-access-plw2s\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.171650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.171144    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963bd392-2b78-4fda-8c15-322725ef0e66-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.177037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.176990    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plw2s\" (UniqueName: \"kubernetes.io/projected/963bd392-2b78-4fda-8c15-322725ef0e66-kube-api-access-plw2s\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.347577 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.347488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:31.398528 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.398475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"]
Sep 16 19:07:31.401503 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:07:31.401360    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af3e11a_bcb6_4605_9abf_f5a53a0f41d3.slice/crio-9110959b2a84ab6066fa4e57f78b6159016dc856004ddbced137ab833e4f4891 WatchSource:0}: Error finding container 9110959b2a84ab6066fa4e57f78b6159016dc856004ddbced137ab833e4f4891: Status 404 returned error can't find the container with id 9110959b2a84ab6066fa4e57f78b6159016dc856004ddbced137ab833e4f4891
Sep 16 19:07:31.563529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.563494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" event={"ID":"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3","Type":"ContainerStarted","Data":"2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"}
Sep 16 19:07:31.563689 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.563548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" event={"ID":"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3","Type":"ContainerStarted","Data":"9110959b2a84ab6066fa4e57f78b6159016dc856004ddbced137ab833e4f4891"}
Sep 16 19:07:31.581472 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.581430    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" podStartSLOduration=1.581415478 podStartE2EDuration="1.581415478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:31.581064452 +0000 UTC m=+1352.885081430" watchObservedRunningTime="2026-09-16 19:07:31.581415478 +0000 UTC m=+1352.885432444"
Sep 16 19:07:31.620702 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:31.620681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"]
Sep 16 19:07:31.622057 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:07:31.622034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963bd392_2b78_4fda_8c15_322725ef0e66.slice/crio-96fe826a1e7e24cdc840d575cfb914c55bf43c04ebcb4cc26757c6ebc2629a60 WatchSource:0}: Error finding container 96fe826a1e7e24cdc840d575cfb914c55bf43c04ebcb4cc26757c6ebc2629a60: Status 404 returned error can't find the container with id 96fe826a1e7e24cdc840d575cfb914c55bf43c04ebcb4cc26757c6ebc2629a60
Sep 16 19:07:32.119809 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:32.119777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:32.570353 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:32.570315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" event={"ID":"963bd392-2b78-4fda-8c15-322725ef0e66","Type":"ContainerStarted","Data":"4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"}
Sep 16 19:07:32.570542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:32.570378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" event={"ID":"963bd392-2b78-4fda-8c15-322725ef0e66","Type":"ContainerStarted","Data":"96fe826a1e7e24cdc840d575cfb914c55bf43c04ebcb4cc26757c6ebc2629a60"}
Sep 16 19:07:32.570542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:32.570410    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:07:32.599673 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:32.599623    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" podStartSLOduration=1.5996090760000001 podStartE2EDuration="1.599609076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:32.598222525 +0000 UTC m=+1353.902239490" watchObservedRunningTime="2026-09-16 19:07:32.599609076 +0000 UTC m=+1353.903626043"
Sep 16 19:07:41.119143 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:41.119105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:41.131793 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:41.131768    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:07:41.625023 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:07:41.624995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:08:03.576591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:03.576559    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:08:05.223696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.223656    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"]
Sep 16 19:08:05.230218 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.230183    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"]
Sep 16 19:08:05.233546 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:08:05.233503    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:08:05.233546 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:05.233546 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:05.233546 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93" cmd=["/bin/sleep","15"]
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.233559    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" message="" err=<
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:08:05.233577    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" podUID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" containerName="main" containerID="cri-o://2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"
Sep 16 19:08:05.233881 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.233630    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" podUID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" containerName="main" containerID="cri-o://2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93" gracePeriod=60
Sep 16 19:08:05.585811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.585787    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:08:05.646303 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tls-certs\") pod \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") "
Sep 16 19:08:05.646522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-dshm\") pod \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") "
Sep 16 19:08:05.646522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tmp-dir\") pod \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") "
Sep 16 19:08:05.646643 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-model-cache\") pod \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") "
Sep 16 19:08:05.646643 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-home\") pod \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") "
Sep 16 19:08:05.646643 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" (UID: "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:05.646643 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646640    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fct6\" (UniqueName: \"kubernetes.io/projected/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-kube-api-access-4fct6\") pod \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\" (UID: \"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3\") "
Sep 16 19:08:05.646828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" (UID: "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:05.646828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-home" (OuterVolumeSpecName: "home") pod "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" (UID: "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:05.646974 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646946    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:05.646974 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646966    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:05.646974 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.646974    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:05.648631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.648609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-dshm" (OuterVolumeSpecName: "dshm") pod "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" (UID: "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:05.648716 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.648659    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" (UID: "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:05.648716 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.648668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-kube-api-access-4fct6" (OuterVolumeSpecName: "kube-api-access-4fct6") pod "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" (UID: "7af3e11a-bcb6-4605-9abf-f5a53a0f41d3"). InnerVolumeSpecName "kube-api-access-4fct6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:05.730247 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.730214    2581 generic.go:356] "Generic (PLEG): container finished" podID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" containerID="2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93" exitCode=0
Sep 16 19:08:05.730417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.730259    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"
Sep 16 19:08:05.730417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.730301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" event={"ID":"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3","Type":"ContainerDied","Data":"2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"}
Sep 16 19:08:05.730417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.730357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn" event={"ID":"7af3e11a-bcb6-4605-9abf-f5a53a0f41d3","Type":"ContainerDied","Data":"9110959b2a84ab6066fa4e57f78b6159016dc856004ddbced137ab833e4f4891"}
Sep 16 19:08:05.730417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.730382    2581 scope.go:117] "RemoveContainer" containerID="2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"
Sep 16 19:08:05.739649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.739632    2581 scope.go:117] "RemoveContainer" containerID="2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"
Sep 16 19:08:05.740031 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:08:05.740003    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93\": container with ID starting with 2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93 not found: ID does not exist" containerID="2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"
Sep 16 19:08:05.740077 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.740043    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93"} err="failed to get container status \"2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93\": rpc error: code = NotFound desc = could not find container \"2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93\": container with ID starting with 2f6454a2da7412a2e8319066a082bc2e2e4a1cdcb2c27b28955a69acdf623c93 not found: ID does not exist"
Sep 16 19:08:05.748095 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.748074    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:05.748095 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.748094    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fct6\" (UniqueName: \"kubernetes.io/projected/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-kube-api-access-4fct6\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:05.748266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.748104    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:05.756139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.756120    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"]
Sep 16 19:08:05.758780 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:05.758757    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-jkxbn"]
Sep 16 19:08:07.222095 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:07.222061    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" path="/var/lib/kubelet/pods/7af3e11a-bcb6-4605-9abf-f5a53a0f41d3/volumes"
Sep 16 19:08:13.577103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:13.577059    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:08:15.903189 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:15.903156    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 19:08:15.903451 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:08:15.903195    2581 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" containerName="main" containerID="cri-o://4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"
Sep 16 19:08:15.903451 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:15.903236    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" containerName="main" containerID="cri-o://4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9" gracePeriod=50
Sep 16 19:08:16.040579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.040555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx_963bd392-2b78-4fda-8c15-322725ef0e66/main/0.log"
Sep 16 19:08:16.042879 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.042848    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:08:16.134439 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.134408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963bd392-2b78-4fda-8c15-322725ef0e66-tls-certs\") pod \"963bd392-2b78-4fda-8c15-322725ef0e66\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") "
Sep 16 19:08:16.134588 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.134506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plw2s\" (UniqueName: \"kubernetes.io/projected/963bd392-2b78-4fda-8c15-322725ef0e66-kube-api-access-plw2s\") pod \"963bd392-2b78-4fda-8c15-322725ef0e66\" (UID: \"963bd392-2b78-4fda-8c15-322725ef0e66\") "
Sep 16 19:08:16.136599 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.136566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963bd392-2b78-4fda-8c15-322725ef0e66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "963bd392-2b78-4fda-8c15-322725ef0e66" (UID: "963bd392-2b78-4fda-8c15-322725ef0e66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:16.136726 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.136573    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963bd392-2b78-4fda-8c15-322725ef0e66-kube-api-access-plw2s" (OuterVolumeSpecName: "kube-api-access-plw2s") pod "963bd392-2b78-4fda-8c15-322725ef0e66" (UID: "963bd392-2b78-4fda-8c15-322725ef0e66"). InnerVolumeSpecName "kube-api-access-plw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:16.235909 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.235814    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plw2s\" (UniqueName: \"kubernetes.io/projected/963bd392-2b78-4fda-8c15-322725ef0e66-kube-api-access-plw2s\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:16.235909 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.235838    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/963bd392-2b78-4fda-8c15-322725ef0e66-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:08:16.784685 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.784657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx_963bd392-2b78-4fda-8c15-322725ef0e66/main/0.log"
Sep 16 19:08:16.784893 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.784698    2581 generic.go:356] "Generic (PLEG): container finished" podID="963bd392-2b78-4fda-8c15-322725ef0e66" containerID="4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9" exitCode=1
Sep 16 19:08:16.784893 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.784728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" event={"ID":"963bd392-2b78-4fda-8c15-322725ef0e66","Type":"ContainerDied","Data":"4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"}
Sep 16 19:08:16.784893 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.784750    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"
Sep 16 19:08:16.784893 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.784786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx" event={"ID":"963bd392-2b78-4fda-8c15-322725ef0e66","Type":"ContainerDied","Data":"96fe826a1e7e24cdc840d575cfb914c55bf43c04ebcb4cc26757c6ebc2629a60"}
Sep 16 19:08:16.784893 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.784811    2581 scope.go:117] "RemoveContainer" containerID="4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"
Sep 16 19:08:16.793613 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.793597    2581 scope.go:117] "RemoveContainer" containerID="4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"
Sep 16 19:08:16.793876 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:08:16.793845    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9\": container with ID starting with 4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9 not found: ID does not exist" containerID="4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"
Sep 16 19:08:16.793950 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.793882    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9"} err="failed to get container status \"4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9\": rpc error: code = NotFound desc = could not find container \"4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9\": container with ID starting with 4e883ffc96f67809b6cb03fe1c77e6437745c335d2d8ca1f1fe7dea22d3593f9 not found: ID does not exist"
Sep 16 19:08:16.807136 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.807115    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"]
Sep 16 19:08:16.810926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:16.810906    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5qlgxx"]
Sep 16 19:08:17.221132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:17.221098    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" path="/var/lib/kubelet/pods/963bd392-2b78-4fda-8c15-322725ef0e66/volumes"
Sep 16 19:08:59.304747 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:08:59.304675    2581 scope.go:117] "RemoveContainer" containerID="4d4f882fbf640dbc46c77db065fc8d1649a7c5f9966eac31b3c2b23cbd7c2852"
Sep 16 19:09:11.607826 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.607791    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"]
Sep 16 19:09:11.608244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.608155    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" containerName="main"
Sep 16 19:09:11.608244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.608168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" containerName="main"
Sep 16 19:09:11.608244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.608177    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" containerName="main"
Sep 16 19:09:11.608244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.608184    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" containerName="main"
Sep 16 19:09:11.608244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.608242    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af3e11a-bcb6-4605-9abf-f5a53a0f41d3" containerName="main"
Sep 16 19:09:11.608407 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.608253    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="963bd392-2b78-4fda-8c15-322725ef0e66" containerName="main"
Sep 16 19:09:11.611107 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.611090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.613926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.613908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:09:11.615129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.615098    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:09:11.615129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.615115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-nk7qd\""
Sep 16 19:09:11.615297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.615196    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:09:11.620569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.620545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"]
Sep 16 19:09:11.678911 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.678852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.679072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.678922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.679072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.678954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.679072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.678971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f91578-3d65-47f8-862d-93f48c929da2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.679072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.679038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.679213 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.679072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvhv\" (UniqueName: \"kubernetes.io/projected/38f91578-3d65-47f8-862d-93f48c929da2-kube-api-access-9zvhv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780252    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f91578-3d65-47f8-862d-93f48c929da2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780665 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvhv\" (UniqueName: \"kubernetes.io/projected/38f91578-3d65-47f8-862d-93f48c929da2-kube-api-access-9zvhv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780665 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780665 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780652    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.780848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.780827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.782574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.782552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.782758 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.782742    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f91578-3d65-47f8-862d-93f48c929da2-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.789352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.789326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvhv\" (UniqueName: \"kubernetes.io/projected/38f91578-3d65-47f8-862d-93f48c929da2-kube-api-access-9zvhv\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-5ttb9\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.867229 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.867147    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"]
Sep 16 19:09:11.870714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.870693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:11.873726 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.873703    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-jsr8q\""
Sep 16 19:09:11.884330 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.884310    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"]
Sep 16 19:09:11.921401 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.921370    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:11.982456 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.982416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:11.982588 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.982564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d876b41-0cae-4258-9322-1fdc64a62431-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:11.982653 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.982634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:11.982693 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:11.982666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qfcq\" (UniqueName: \"kubernetes.io/projected/0d876b41-0cae-4258-9322-1fdc64a62431-kube-api-access-5qfcq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.083599 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.083552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.083816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.083689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d876b41-0cae-4258-9322-1fdc64a62431-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.083816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.083754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.083816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.083806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qfcq\" (UniqueName: \"kubernetes.io/projected/0d876b41-0cae-4258-9322-1fdc64a62431-kube-api-access-5qfcq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.084062 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.083930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.084244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.084215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.086276 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.086255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d876b41-0cae-4258-9322-1fdc64a62431-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.092498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.092478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qfcq\" (UniqueName: \"kubernetes.io/projected/0d876b41-0cae-4258-9322-1fdc64a62431-kube-api-access-5qfcq\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.182067 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.182040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:12.199126 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.199104    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"]
Sep 16 19:09:12.200928 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:09:12.200902    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f91578_3d65_47f8_862d_93f48c929da2.slice/crio-af3f66bd354e1b1e0683b1f25a5c8eaa75465faab57a0c275afa8a936d944a6a WatchSource:0}: Error finding container af3f66bd354e1b1e0683b1f25a5c8eaa75465faab57a0c275afa8a936d944a6a: Status 404 returned error can't find the container with id af3f66bd354e1b1e0683b1f25a5c8eaa75465faab57a0c275afa8a936d944a6a
Sep 16 19:09:12.462880 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:12.462769    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"]
Sep 16 19:09:12.466184 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:09:12.466145    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d876b41_0cae_4258_9322_1fdc64a62431.slice/crio-237d17ac3749d4ed9bd141d4a8f3176043c6ecbc65efe300ee4dda4ec20f9cf3 WatchSource:0}: Error finding container 237d17ac3749d4ed9bd141d4a8f3176043c6ecbc65efe300ee4dda4ec20f9cf3: Status 404 returned error can't find the container with id 237d17ac3749d4ed9bd141d4a8f3176043c6ecbc65efe300ee4dda4ec20f9cf3
Sep 16 19:09:13.056821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.056778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" event={"ID":"0d876b41-0cae-4258-9322-1fdc64a62431","Type":"ContainerStarted","Data":"497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218"}
Sep 16 19:09:13.057347 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.056839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" event={"ID":"0d876b41-0cae-4258-9322-1fdc64a62431","Type":"ContainerStarted","Data":"237d17ac3749d4ed9bd141d4a8f3176043c6ecbc65efe300ee4dda4ec20f9cf3"}
Sep 16 19:09:13.057347 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.056898    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:13.058484 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.058460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" event={"ID":"38f91578-3d65-47f8-862d-93f48c929da2","Type":"ContainerStarted","Data":"b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"}
Sep 16 19:09:13.058587 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.058497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" event={"ID":"38f91578-3d65-47f8-862d-93f48c929da2","Type":"ContainerStarted","Data":"af3f66bd354e1b1e0683b1f25a5c8eaa75465faab57a0c275afa8a936d944a6a"}
Sep 16 19:09:13.078359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.078315    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" podStartSLOduration=2.078301003 podStartE2EDuration="2.078301003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:13.076171834 +0000 UTC m=+1454.380188839" watchObservedRunningTime="2026-09-16 19:09:13.078301003 +0000 UTC m=+1454.382317970"
Sep 16 19:09:13.095757 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.095718    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" podStartSLOduration=2.095704734 podStartE2EDuration="2.095704734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:13.093895691 +0000 UTC m=+1454.397912660" watchObservedRunningTime="2026-09-16 19:09:13.095704734 +0000 UTC m=+1454.399721701"
Sep 16 19:09:13.922119 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:13.922084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:21.921572 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:21.921535    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:21.933764 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:21.933739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:22.113081 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:22.113048    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:44.065249 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:44.065217    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:09:45.758046 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:45.758006    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"]
Sep 16 19:09:45.767058 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:09:45.767021    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:09:45.767058 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:45.767058 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:45.767058 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9" cmd=["/bin/sleep","15"]
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:45.767075    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" message="" err=<
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:09:45.767088    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" podUID="38f91578-3d65-47f8-862d-93f48c929da2" containerName="main" containerID="cri-o://b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"
Sep 16 19:09:45.767321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:45.767135    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" podUID="38f91578-3d65-47f8-862d-93f48c929da2" containerName="main" containerID="cri-o://b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9" gracePeriod=60
Sep 16 19:09:45.772419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:45.772392    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"]
Sep 16 19:09:46.116529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.116507    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:46.160465 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160433    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f91578-3d65-47f8-862d-93f48c929da2-tls-certs\") pod \"38f91578-3d65-47f8-862d-93f48c929da2\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") "
Sep 16 19:09:46.160604 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160475    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-tmp-dir\") pod \"38f91578-3d65-47f8-862d-93f48c929da2\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") "
Sep 16 19:09:46.160604 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-dshm\") pod \"38f91578-3d65-47f8-862d-93f48c929da2\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") "
Sep 16 19:09:46.160695 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-home\") pod \"38f91578-3d65-47f8-862d-93f48c929da2\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") "
Sep 16 19:09:46.160695 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-model-cache\") pod \"38f91578-3d65-47f8-862d-93f48c929da2\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") "
Sep 16 19:09:46.160695 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160669    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zvhv\" (UniqueName: \"kubernetes.io/projected/38f91578-3d65-47f8-862d-93f48c929da2-kube-api-access-9zvhv\") pod \"38f91578-3d65-47f8-862d-93f48c929da2\" (UID: \"38f91578-3d65-47f8-862d-93f48c929da2\") "
Sep 16 19:09:46.160871 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38f91578-3d65-47f8-862d-93f48c929da2" (UID: "38f91578-3d65-47f8-862d-93f48c929da2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:46.160940 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-home" (OuterVolumeSpecName: "home") pod "38f91578-3d65-47f8-862d-93f48c929da2" (UID: "38f91578-3d65-47f8-862d-93f48c929da2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:46.160940 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.160916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-model-cache" (OuterVolumeSpecName: "model-cache") pod "38f91578-3d65-47f8-862d-93f48c929da2" (UID: "38f91578-3d65-47f8-862d-93f48c929da2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:46.161041 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.161027    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:46.161096 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.161047    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:46.161096 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.161060    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:46.162786 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.162766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38f91578-3d65-47f8-862d-93f48c929da2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38f91578-3d65-47f8-862d-93f48c929da2" (UID: "38f91578-3d65-47f8-862d-93f48c929da2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:46.162786 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.162776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-dshm" (OuterVolumeSpecName: "dshm") pod "38f91578-3d65-47f8-862d-93f48c929da2" (UID: "38f91578-3d65-47f8-862d-93f48c929da2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:46.163123 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.163108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38f91578-3d65-47f8-862d-93f48c929da2-kube-api-access-9zvhv" (OuterVolumeSpecName: "kube-api-access-9zvhv") pod "38f91578-3d65-47f8-862d-93f48c929da2" (UID: "38f91578-3d65-47f8-862d-93f48c929da2"). InnerVolumeSpecName "kube-api-access-9zvhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:46.220075 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.220043    2581 generic.go:356] "Generic (PLEG): container finished" podID="38f91578-3d65-47f8-862d-93f48c929da2" containerID="b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9" exitCode=0
Sep 16 19:09:46.220177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.220087    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"
Sep 16 19:09:46.220177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.220130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" event={"ID":"38f91578-3d65-47f8-862d-93f48c929da2","Type":"ContainerDied","Data":"b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"}
Sep 16 19:09:46.220177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.220173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" event={"ID":"38f91578-3d65-47f8-862d-93f48c929da2","Type":"ContainerDied","Data":"af3f66bd354e1b1e0683b1f25a5c8eaa75465faab57a0c275afa8a936d944a6a"}
Sep 16 19:09:46.220287 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.220188    2581 scope.go:117] "RemoveContainer" containerID="b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"
Sep 16 19:09:46.229135 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.229116    2581 scope.go:117] "RemoveContainer" containerID="b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"
Sep 16 19:09:46.229391 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:09:46.229374    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9\": container with ID starting with b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9 not found: ID does not exist" containerID="b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"
Sep 16 19:09:46.229440 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.229398    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9"} err="failed to get container status \"b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9\": rpc error: code = NotFound desc = could not find container \"b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9\": container with ID starting with b6e208e1b195d2ba5fd579b4d39d267e8e8435ad901ba29a0441f26fae0b62d9 not found: ID does not exist"
Sep 16 19:09:46.243113 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.243088    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"]
Sep 16 19:09:46.247368 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.247348    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9"]
Sep 16 19:09:46.261419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.261403    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38f91578-3d65-47f8-862d-93f48c929da2-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:46.261419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.261420    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38f91578-3d65-47f8-862d-93f48c929da2-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:46.261530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:46.261429    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvhv\" (UniqueName: \"kubernetes.io/projected/38f91578-3d65-47f8-862d-93f48c929da2-kube-api-access-9zvhv\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:09:47.103082 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:47.103038    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-5ttb9" podUID="38f91578-3d65-47f8-862d-93f48c929da2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:09:47.222630 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:47.222593    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38f91578-3d65-47f8-862d-93f48c929da2" path="/var/lib/kubelet/pods/38f91578-3d65-47f8-862d-93f48c929da2/volumes"
Sep 16 19:09:54.065507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:54.065470    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:59.263670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:59.263640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:09:59.264246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:59.264226    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:09:59.266637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:59.266618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:09:59.267144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:09:59.267126    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:10:00.791746 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:00.791703    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main" containerID="cri-o://497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218" gracePeriod=45
Sep 16 19:10:04.065385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:04.065348    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:14.064896 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:14.064835    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:14.065359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:14.064964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:10:24.064728 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:24.064644    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:10:31.032895 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.032844    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:10:31.110935 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.110848    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-tokenizer-cache\") pod \"0d876b41-0cae-4258-9322-1fdc64a62431\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") "
Sep 16 19:10:31.110935 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.110904    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qfcq\" (UniqueName: \"kubernetes.io/projected/0d876b41-0cae-4258-9322-1fdc64a62431-kube-api-access-5qfcq\") pod \"0d876b41-0cae-4258-9322-1fdc64a62431\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") "
Sep 16 19:10:31.111141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.110937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-cachi2-cache\") pod \"0d876b41-0cae-4258-9322-1fdc64a62431\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") "
Sep 16 19:10:31.111141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.110975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d876b41-0cae-4258-9322-1fdc64a62431-tls-certs\") pod \"0d876b41-0cae-4258-9322-1fdc64a62431\" (UID: \"0d876b41-0cae-4258-9322-1fdc64a62431\") "
Sep 16 19:10:31.111258 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.111149    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "0d876b41-0cae-4258-9322-1fdc64a62431" (UID: "0d876b41-0cae-4258-9322-1fdc64a62431"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:31.111258 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.111228    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "0d876b41-0cae-4258-9322-1fdc64a62431" (UID: "0d876b41-0cae-4258-9322-1fdc64a62431"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:10:31.113007 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.112982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d876b41-0cae-4258-9322-1fdc64a62431-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d876b41-0cae-4258-9322-1fdc64a62431" (UID: "0d876b41-0cae-4258-9322-1fdc64a62431"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:10:31.113085 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.113066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d876b41-0cae-4258-9322-1fdc64a62431-kube-api-access-5qfcq" (OuterVolumeSpecName: "kube-api-access-5qfcq") pod "0d876b41-0cae-4258-9322-1fdc64a62431" (UID: "0d876b41-0cae-4258-9322-1fdc64a62431"). InnerVolumeSpecName "kube-api-access-5qfcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:10:31.212274 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.212240    2581 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-cachi2-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:10:31.212274 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.212270    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d876b41-0cae-4258-9322-1fdc64a62431-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:10:31.212274 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.212280    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d876b41-0cae-4258-9322-1fdc64a62431-tokenizer-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:10:31.212453 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.212289    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qfcq\" (UniqueName: \"kubernetes.io/projected/0d876b41-0cae-4258-9322-1fdc64a62431-kube-api-access-5qfcq\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:10:31.436395 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.436366    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d876b41-0cae-4258-9322-1fdc64a62431" containerID="497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218" exitCode=0
Sep 16 19:10:31.436550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.436426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" event={"ID":"0d876b41-0cae-4258-9322-1fdc64a62431","Type":"ContainerDied","Data":"497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218"}
Sep 16 19:10:31.436550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.436430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"
Sep 16 19:10:31.436550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.436466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f" event={"ID":"0d876b41-0cae-4258-9322-1fdc64a62431","Type":"ContainerDied","Data":"237d17ac3749d4ed9bd141d4a8f3176043c6ecbc65efe300ee4dda4ec20f9cf3"}
Sep 16 19:10:31.436550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.436487    2581 scope.go:117] "RemoveContainer" containerID="497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218"
Sep 16 19:10:31.444979 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.444961    2581 scope.go:117] "RemoveContainer" containerID="497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218"
Sep 16 19:10:31.445258 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:10:31.445234    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218\": container with ID starting with 497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218 not found: ID does not exist" containerID="497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218"
Sep 16 19:10:31.445331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.445270    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218"} err="failed to get container status \"497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218\": rpc error: code = NotFound desc = could not find container \"497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218\": container with ID starting with 497631beb121cbdc5e93cca4428604eafe87da8d92047b41293e35ea5f606218 not found: ID does not exist"
Sep 16 19:10:31.454589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.454568    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"]
Sep 16 19:10:31.458126 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:31.458108    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcscs5f"]
Sep 16 19:10:33.221452 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:33.221418    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" path="/var/lib/kubelet/pods/0d876b41-0cae-4258-9322-1fdc64a62431/volumes"
Sep 16 19:10:47.313066 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313015    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"]
Sep 16 19:10:47.313571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313398    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38f91578-3d65-47f8-862d-93f48c929da2" containerName="main"
Sep 16 19:10:47.313571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313414    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38f91578-3d65-47f8-862d-93f48c929da2" containerName="main"
Sep 16 19:10:47.313571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313439    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main"
Sep 16 19:10:47.313571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313445    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main"
Sep 16 19:10:47.313571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313502    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d876b41-0cae-4258-9322-1fdc64a62431" containerName="main"
Sep 16 19:10:47.313571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.313511    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38f91578-3d65-47f8-862d-93f48c929da2" containerName="main"
Sep 16 19:10:47.318181 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.318165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.322872 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.322815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 19:10:47.322989 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.322882    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 19:10:47.323226 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.323056    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 19:10:47.323226 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.323142    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-xvjmh\""
Sep 16 19:10:47.323422 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.323399    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-dq4rh\""
Sep 16 19:10:47.335003 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.334944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"]
Sep 16 19:10:47.337958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.337934    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"]
Sep 16 19:10:47.341597 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.341578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.353745 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.353726    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"]
Sep 16 19:10:47.441166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzlh\" (UniqueName: \"kubernetes.io/projected/e769617e-46c1-425e-8e92-98fa08921c60-kube-api-access-8pzlh\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.441334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.441334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffjgd\" (UniqueName: \"kubernetes.io/projected/7373cc2c-e7f2-4c9d-8035-582d936725ac-kube-api-access-ffjgd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.441334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.441334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e769617e-46c1-425e-8e92-98fa08921c60-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.441334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.441334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373cc2c-e7f2-4c9d-8035-582d936725ac-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.441639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.441639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.441639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.441639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.441639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.441524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.542476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzlh\" (UniqueName: \"kubernetes.io/projected/e769617e-46c1-425e-8e92-98fa08921c60-kube-api-access-8pzlh\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.542628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.542628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffjgd\" (UniqueName: \"kubernetes.io/projected/7373cc2c-e7f2-4c9d-8035-582d936725ac-kube-api-access-ffjgd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.542628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.542628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542595    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e769617e-46c1-425e-8e92-98fa08921c60-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.542628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.542930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373cc2c-e7f2-4c9d-8035-582d936725ac-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.542930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.542930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.543091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.542982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.543091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.543191 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.543245 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.543302 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.543356 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.543409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.543633 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.543633 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.543609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.545817 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.544381    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"]
Sep 16 19:10:47.546023 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.546000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.546098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.546032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.547065 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.546499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373cc2c-e7f2-4c9d-8035-582d936725ac-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.547065 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.547004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e769617e-46c1-425e-8e92-98fa08921c60-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.549297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.549279    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.552037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.552013    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-pvxtn\""
Sep 16 19:10:47.556305 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.556280    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzlh\" (UniqueName: \"kubernetes.io/projected/e769617e-46c1-425e-8e92-98fa08921c60-kube-api-access-8pzlh\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.558635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.558613    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"]
Sep 16 19:10:47.559280 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.559256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffjgd\" (UniqueName: \"kubernetes.io/projected/7373cc2c-e7f2-4c9d-8035-582d936725ac-kube-api-access-ffjgd\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.628455 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.628427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:47.644563 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.644524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db505cc-04e3-406a-8ebc-c75a28754435-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.644677 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.644589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lncn\" (UniqueName: \"kubernetes.io/projected/4db505cc-04e3-406a-8ebc-c75a28754435-kube-api-access-2lncn\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.652433 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.652408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:47.745407 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.745377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db505cc-04e3-406a-8ebc-c75a28754435-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.745407 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.745419    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lncn\" (UniqueName: \"kubernetes.io/projected/4db505cc-04e3-406a-8ebc-c75a28754435-kube-api-access-2lncn\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.748242 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.748215    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db505cc-04e3-406a-8ebc-c75a28754435-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.753682 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.753658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lncn\" (UniqueName: \"kubernetes.io/projected/4db505cc-04e3-406a-8ebc-c75a28754435-kube-api-access-2lncn\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.878115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.878032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:47.907651 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.907624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"]
Sep 16 19:10:47.909837 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:10:47.909810    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode769617e_46c1_425e_8e92_98fa08921c60.slice/crio-b9161309c2d8a32456b5f10302d78b642db054faa3320e94fcb7bf6cff37e4d5 WatchSource:0}: Error finding container b9161309c2d8a32456b5f10302d78b642db054faa3320e94fcb7bf6cff37e4d5: Status 404 returned error can't find the container with id b9161309c2d8a32456b5f10302d78b642db054faa3320e94fcb7bf6cff37e4d5
Sep 16 19:10:47.911901 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.911881    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:10:47.938801 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:47.938751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"]
Sep 16 19:10:47.942822 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:10:47.942790    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7373cc2c_e7f2_4c9d_8035_582d936725ac.slice/crio-6bb76c55d572ee4072cc3835a7ec3a828e0c1b90506934103e5b7b47e9249316 WatchSource:0}: Error finding container 6bb76c55d572ee4072cc3835a7ec3a828e0c1b90506934103e5b7b47e9249316: Status 404 returned error can't find the container with id 6bb76c55d572ee4072cc3835a7ec3a828e0c1b90506934103e5b7b47e9249316
Sep 16 19:10:48.153594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.153483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"]
Sep 16 19:10:48.155756 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:10:48.155728    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db505cc_04e3_406a_8ebc_c75a28754435.slice/crio-4b13155b4381e090fb12ccded4c9fbd0c92744939d107a95d3f1a8dc7dfc44cc WatchSource:0}: Error finding container 4b13155b4381e090fb12ccded4c9fbd0c92744939d107a95d3f1a8dc7dfc44cc: Status 404 returned error can't find the container with id 4b13155b4381e090fb12ccded4c9fbd0c92744939d107a95d3f1a8dc7dfc44cc
Sep 16 19:10:48.518253 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.518176    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" event={"ID":"4db505cc-04e3-406a-8ebc-c75a28754435","Type":"ContainerStarted","Data":"6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3"}
Sep 16 19:10:48.518253 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.518229    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" event={"ID":"4db505cc-04e3-406a-8ebc-c75a28754435","Type":"ContainerStarted","Data":"4b13155b4381e090fb12ccded4c9fbd0c92744939d107a95d3f1a8dc7dfc44cc"}
Sep 16 19:10:48.518729 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.518286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:10:48.519812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.519790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" event={"ID":"7373cc2c-e7f2-4c9d-8035-582d936725ac","Type":"ContainerStarted","Data":"ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"}
Sep 16 19:10:48.519921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.519829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" event={"ID":"7373cc2c-e7f2-4c9d-8035-582d936725ac","Type":"ContainerStarted","Data":"6bb76c55d572ee4072cc3835a7ec3a828e0c1b90506934103e5b7b47e9249316"}
Sep 16 19:10:48.521115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.521094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" event={"ID":"e769617e-46c1-425e-8e92-98fa08921c60","Type":"ContainerStarted","Data":"b9161309c2d8a32456b5f10302d78b642db054faa3320e94fcb7bf6cff37e4d5"}
Sep 16 19:10:48.560510 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.560449    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podStartSLOduration=1.5604215049999999 podStartE2EDuration="1.560421505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:48.557374495 +0000 UTC m=+1549.861391463" watchObservedRunningTime="2026-09-16 19:10:48.560421505 +0000 UTC m=+1549.864438473"
Sep 16 19:10:48.585640 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.585589    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" podStartSLOduration=1.58557364 podStartE2EDuration="1.58557364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:48.582812152 +0000 UTC m=+1549.886829119" watchObservedRunningTime="2026-09-16 19:10:48.58557364 +0000 UTC m=+1549.889590608"
Sep 16 19:10:48.658656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:48.653444    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:49.527288 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:49.527244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" event={"ID":"e769617e-46c1-425e-8e92-98fa08921c60","Type":"ContainerStarted","Data":"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"}
Sep 16 19:10:49.527650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:49.527451    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:50.534457 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:50.534420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" event={"ID":"e769617e-46c1-425e-8e92-98fa08921c60","Type":"ContainerStarted","Data":"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"}
Sep 16 19:10:50.555465 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:50.555418    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" podStartSLOduration=2.087907391 podStartE2EDuration="3.555403235s" totalImagesPullingTime="1.467495844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:50.553798277 +0000 UTC m=+1551.857815284" watchObservedRunningTime="2026-09-16 19:10:50.555403235 +0000 UTC m=+1551.859420201"
Sep 16 19:10:50.629249 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:50.629225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:57.629289 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:57.629254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:57.641645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:57.641618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:57.653060 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:57.653042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:57.665629 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:57.665608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:58.585639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:58.585609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:10:58.586419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:58.586398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:10:59.595585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:10:59.595553    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:11:19.529411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:19.529381    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:11:26.906800 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.906762    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"]
Sep 16 19:11:26.909557 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.909515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"]
Sep 16 19:11:26.912679 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.912652    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"]
Sep 16 19:11:26.916721 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:26.916686    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:26.916721 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:26.916721 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:26.916721 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45" cmd=["/bin/sleep","15"]
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.916749    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" message="" err=<
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:26.916766    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" podUID="7373cc2c-e7f2-4c9d-8035-582d936725ac" containerName="main" containerID="cri-o://ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"
Sep 16 19:11:26.916999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.916819    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" podUID="7373cc2c-e7f2-4c9d-8035-582d936725ac" containerName="main" containerID="cri-o://ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45" gracePeriod=60
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:26.921884    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e" cmd=["/bin/sleep","15"]
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.921928    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" message="" err=<
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:26.921941    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:26.921958 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="main" containerID="cri-o://5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"
Sep 16 19:11:26.922593 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:26.921980    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="main" containerID="cri-o://5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e" gracePeriod=60
Sep 16 19:11:27.138194 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.138159    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="llm-d-routing-sidecar" containerID="cri-o://dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4" gracePeriod=60
Sep 16 19:11:27.255778 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.255752    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:11:27.360369 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-dshm\") pod \"7373cc2c-e7f2-4c9d-8035-582d936725ac\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") "
Sep 16 19:11:27.360533 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-tmp-dir\") pod \"7373cc2c-e7f2-4c9d-8035-582d936725ac\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") "
Sep 16 19:11:27.360533 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373cc2c-e7f2-4c9d-8035-582d936725ac-tls-certs\") pod \"7373cc2c-e7f2-4c9d-8035-582d936725ac\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") "
Sep 16 19:11:27.360533 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-model-cache\") pod \"7373cc2c-e7f2-4c9d-8035-582d936725ac\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") "
Sep 16 19:11:27.360533 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-home\") pod \"7373cc2c-e7f2-4c9d-8035-582d936725ac\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") "
Sep 16 19:11:27.360768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffjgd\" (UniqueName: \"kubernetes.io/projected/7373cc2c-e7f2-4c9d-8035-582d936725ac-kube-api-access-ffjgd\") pod \"7373cc2c-e7f2-4c9d-8035-582d936725ac\" (UID: \"7373cc2c-e7f2-4c9d-8035-582d936725ac\") "
Sep 16 19:11:27.360768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7373cc2c-e7f2-4c9d-8035-582d936725ac" (UID: "7373cc2c-e7f2-4c9d-8035-582d936725ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.360768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "7373cc2c-e7f2-4c9d-8035-582d936725ac" (UID: "7373cc2c-e7f2-4c9d-8035-582d936725ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.360947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360798    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-home" (OuterVolumeSpecName: "home") pod "7373cc2c-e7f2-4c9d-8035-582d936725ac" (UID: "7373cc2c-e7f2-4c9d-8035-582d936725ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.360947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360880    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.360947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360901    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.360947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.360914    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.363272 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.363249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-dshm" (OuterVolumeSpecName: "dshm") pod "7373cc2c-e7f2-4c9d-8035-582d936725ac" (UID: "7373cc2c-e7f2-4c9d-8035-582d936725ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.363352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.363305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7373cc2c-e7f2-4c9d-8035-582d936725ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7373cc2c-e7f2-4c9d-8035-582d936725ac" (UID: "7373cc2c-e7f2-4c9d-8035-582d936725ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:27.363394 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.363380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7373cc2c-e7f2-4c9d-8035-582d936725ac-kube-api-access-ffjgd" (OuterVolumeSpecName: "kube-api-access-ffjgd") pod "7373cc2c-e7f2-4c9d-8035-582d936725ac" (UID: "7373cc2c-e7f2-4c9d-8035-582d936725ac"). InnerVolumeSpecName "kube-api-access-ffjgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:27.389157 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.389136    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:11:27.461385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461303    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-home\") pod \"e769617e-46c1-425e-8e92-98fa08921c60\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") "
Sep 16 19:11:27.461385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-dshm\") pod \"e769617e-46c1-425e-8e92-98fa08921c60\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") "
Sep 16 19:11:27.461592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pzlh\" (UniqueName: \"kubernetes.io/projected/e769617e-46c1-425e-8e92-98fa08921c60-kube-api-access-8pzlh\") pod \"e769617e-46c1-425e-8e92-98fa08921c60\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") "
Sep 16 19:11:27.461592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-model-cache\") pod \"e769617e-46c1-425e-8e92-98fa08921c60\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") "
Sep 16 19:11:27.461592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e769617e-46c1-425e-8e92-98fa08921c60-tls-certs\") pod \"e769617e-46c1-425e-8e92-98fa08921c60\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") "
Sep 16 19:11:27.461592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-tmp-dir\") pod \"e769617e-46c1-425e-8e92-98fa08921c60\" (UID: \"e769617e-46c1-425e-8e92-98fa08921c60\") "
Sep 16 19:11:27.461786 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461611    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-home" (OuterVolumeSpecName: "home") pod "e769617e-46c1-425e-8e92-98fa08921c60" (UID: "e769617e-46c1-425e-8e92-98fa08921c60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.461786 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461726    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-model-cache" (OuterVolumeSpecName: "model-cache") pod "e769617e-46c1-425e-8e92-98fa08921c60" (UID: "e769617e-46c1-425e-8e92-98fa08921c60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.461914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461820    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.461914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461832    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.461914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461841    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7373cc2c-e7f2-4c9d-8035-582d936725ac-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.461914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461849    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7373cc2c-e7f2-4c9d-8035-582d936725ac-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.461914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461884    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffjgd\" (UniqueName: \"kubernetes.io/projected/7373cc2c-e7f2-4c9d-8035-582d936725ac-kube-api-access-ffjgd\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.461914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.461829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e769617e-46c1-425e-8e92-98fa08921c60" (UID: "e769617e-46c1-425e-8e92-98fa08921c60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.463545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.463523    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-dshm" (OuterVolumeSpecName: "dshm") pod "e769617e-46c1-425e-8e92-98fa08921c60" (UID: "e769617e-46c1-425e-8e92-98fa08921c60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:27.463545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.463530    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e769617e-46c1-425e-8e92-98fa08921c60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e769617e-46c1-425e-8e92-98fa08921c60" (UID: "e769617e-46c1-425e-8e92-98fa08921c60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:27.463545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.463540    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e769617e-46c1-425e-8e92-98fa08921c60-kube-api-access-8pzlh" (OuterVolumeSpecName: "kube-api-access-8pzlh") pod "e769617e-46c1-425e-8e92-98fa08921c60" (UID: "e769617e-46c1-425e-8e92-98fa08921c60"). InnerVolumeSpecName "kube-api-access-8pzlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:27.562331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.562297    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pzlh\" (UniqueName: \"kubernetes.io/projected/e769617e-46c1-425e-8e92-98fa08921c60-kube-api-access-8pzlh\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.562331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.562329    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e769617e-46c1-425e-8e92-98fa08921c60-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.562331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.562338    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.562331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.562345    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e769617e-46c1-425e-8e92-98fa08921c60-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:11:27.722113 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.722028    2581 generic.go:356] "Generic (PLEG): container finished" podID="7373cc2c-e7f2-4c9d-8035-582d936725ac" containerID="ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45" exitCode=0
Sep 16 19:11:27.722113 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.722070    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"
Sep 16 19:11:27.722391 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.722109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" event={"ID":"7373cc2c-e7f2-4c9d-8035-582d936725ac","Type":"ContainerDied","Data":"ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"}
Sep 16 19:11:27.722391 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.722172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv" event={"ID":"7373cc2c-e7f2-4c9d-8035-582d936725ac","Type":"ContainerDied","Data":"6bb76c55d572ee4072cc3835a7ec3a828e0c1b90506934103e5b7b47e9249316"}
Sep 16 19:11:27.722391 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.722204    2581 scope.go:117] "RemoveContainer" containerID="ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"
Sep 16 19:11:27.724012 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.723992    2581 generic.go:356] "Generic (PLEG): container finished" podID="e769617e-46c1-425e-8e92-98fa08921c60" containerID="5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e" exitCode=0
Sep 16 19:11:27.724012 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.724009    2581 generic.go:356] "Generic (PLEG): container finished" podID="e769617e-46c1-425e-8e92-98fa08921c60" containerID="dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4" exitCode=0
Sep 16 19:11:27.724149 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.724039    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"
Sep 16 19:11:27.724149 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.724077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" event={"ID":"e769617e-46c1-425e-8e92-98fa08921c60","Type":"ContainerDied","Data":"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"}
Sep 16 19:11:27.724149 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.724132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" event={"ID":"e769617e-46c1-425e-8e92-98fa08921c60","Type":"ContainerDied","Data":"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"}
Sep 16 19:11:27.724246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.724159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44" event={"ID":"e769617e-46c1-425e-8e92-98fa08921c60","Type":"ContainerDied","Data":"b9161309c2d8a32456b5f10302d78b642db054faa3320e94fcb7bf6cff37e4d5"}
Sep 16 19:11:27.732455 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.732440    2581 scope.go:117] "RemoveContainer" containerID="ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"
Sep 16 19:11:27.732749 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:27.732723    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45\": container with ID starting with ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45 not found: ID does not exist" containerID="ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"
Sep 16 19:11:27.732881 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.732757    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45"} err="failed to get container status \"ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45\": rpc error: code = NotFound desc = could not find container \"ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45\": container with ID starting with ea1c33bcb3dea0609b0bc463536116a251e8121f06dc6de668119f86214a5d45 not found: ID does not exist"
Sep 16 19:11:27.732881 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.732777    2581 scope.go:117] "RemoveContainer" containerID="5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"
Sep 16 19:11:27.740851 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.740831    2581 scope.go:117] "RemoveContainer" containerID="dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"
Sep 16 19:11:27.746218 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.746192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"]
Sep 16 19:11:27.750675 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.750657    2581 scope.go:117] "RemoveContainer" containerID="5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"
Sep 16 19:11:27.751071 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751048    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-l5scv"]
Sep 16 19:11:27.751174 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:27.751100    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e\": container with ID starting with 5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e not found: ID does not exist" containerID="5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"
Sep 16 19:11:27.751174 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751151    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"} err="failed to get container status \"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e\": rpc error: code = NotFound desc = could not find container \"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e\": container with ID starting with 5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e not found: ID does not exist"
Sep 16 19:11:27.751174 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751168    2581 scope.go:117] "RemoveContainer" containerID="dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"
Sep 16 19:11:27.751446 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:11:27.751426    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4\": container with ID starting with dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4 not found: ID does not exist" containerID="dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"
Sep 16 19:11:27.751509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751451    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"} err="failed to get container status \"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4\": rpc error: code = NotFound desc = could not find container \"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4\": container with ID starting with dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4 not found: ID does not exist"
Sep 16 19:11:27.751509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751464    2581 scope.go:117] "RemoveContainer" containerID="5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"
Sep 16 19:11:27.751753 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751733    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e"} err="failed to get container status \"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e\": rpc error: code = NotFound desc = could not find container \"5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e\": container with ID starting with 5e442af1e7f36b7ba85ad27eccc30a266a2e7ef80cbd8879a431f0c62e77f11e not found: ID does not exist"
Sep 16 19:11:27.751810 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.751756    2581 scope.go:117] "RemoveContainer" containerID="dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"
Sep 16 19:11:27.752092 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.752072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4"} err="failed to get container status \"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4\": rpc error: code = NotFound desc = could not find container \"dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4\": container with ID starting with dcfc253600846cef532fc4fd821527868f3135f61ee9717b13b2c341adefc7b4 not found: ID does not exist"
Sep 16 19:11:27.762378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.762349    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"]
Sep 16 19:11:27.766758 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:27.766739    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-jnh44"]
Sep 16 19:11:29.221423 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:29.221389    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7373cc2c-e7f2-4c9d-8035-582d936725ac" path="/var/lib/kubelet/pods/7373cc2c-e7f2-4c9d-8035-582d936725ac/volumes"
Sep 16 19:11:29.221900 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:29.221768    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e769617e-46c1-425e-8e92-98fa08921c60" path="/var/lib/kubelet/pods/e769617e-46c1-425e-8e92-98fa08921c60/volumes"
Sep 16 19:11:29.528993 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:29.528913    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:39.529094 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:39.529059    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:41.938675 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:41.938631    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main" containerID="cri-o://6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3" gracePeriod=45
Sep 16 19:11:49.528949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:49.528906    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:11:49.529329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:49.529012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:11:59.529018 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:11:59.528982    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:12:09.529147 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:09.529107    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:12:12.184048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.184024    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:12:12.207349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.207288    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lncn\" (UniqueName: \"kubernetes.io/projected/4db505cc-04e3-406a-8ebc-c75a28754435-kube-api-access-2lncn\") pod \"4db505cc-04e3-406a-8ebc-c75a28754435\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") "
Sep 16 19:12:12.207454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.207350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db505cc-04e3-406a-8ebc-c75a28754435-tls-certs\") pod \"4db505cc-04e3-406a-8ebc-c75a28754435\" (UID: \"4db505cc-04e3-406a-8ebc-c75a28754435\") "
Sep 16 19:12:12.209437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.209409    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db505cc-04e3-406a-8ebc-c75a28754435-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4db505cc-04e3-406a-8ebc-c75a28754435" (UID: "4db505cc-04e3-406a-8ebc-c75a28754435"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:12.209531 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.209459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db505cc-04e3-406a-8ebc-c75a28754435-kube-api-access-2lncn" (OuterVolumeSpecName: "kube-api-access-2lncn") pod "4db505cc-04e3-406a-8ebc-c75a28754435" (UID: "4db505cc-04e3-406a-8ebc-c75a28754435"). InnerVolumeSpecName "kube-api-access-2lncn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:12.308344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.308304    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lncn\" (UniqueName: \"kubernetes.io/projected/4db505cc-04e3-406a-8ebc-c75a28754435-kube-api-access-2lncn\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:12.308344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.308337    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db505cc-04e3-406a-8ebc-c75a28754435-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:12.946522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.946490    2581 generic.go:356] "Generic (PLEG): container finished" podID="4db505cc-04e3-406a-8ebc-c75a28754435" containerID="6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3" exitCode=0
Sep 16 19:12:12.946714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.946525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" event={"ID":"4db505cc-04e3-406a-8ebc-c75a28754435","Type":"ContainerDied","Data":"6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3"}
Sep 16 19:12:12.946714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.946552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"
Sep 16 19:12:12.946714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.946678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks" event={"ID":"4db505cc-04e3-406a-8ebc-c75a28754435","Type":"ContainerDied","Data":"4b13155b4381e090fb12ccded4c9fbd0c92744939d107a95d3f1a8dc7dfc44cc"}
Sep 16 19:12:12.946714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.946710    2581 scope.go:117] "RemoveContainer" containerID="6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3"
Sep 16 19:12:12.957849 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.957803    2581 scope.go:117] "RemoveContainer" containerID="6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3"
Sep 16 19:12:12.958276 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:12:12.958248    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3\": container with ID starting with 6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3 not found: ID does not exist" containerID="6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3"
Sep 16 19:12:12.958382 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.958286    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3"} err="failed to get container status \"6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3\": rpc error: code = NotFound desc = could not find container \"6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3\": container with ID starting with 6b2f0a9a79e46100363c83a6792c2b0a43957c83cee8ae5ade08a7ef010a02b3 not found: ID does not exist"
Sep 16 19:12:12.969561 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.969539    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"]
Sep 16 19:12:12.972362 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:12.972343    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75mt8ks"]
Sep 16 19:12:13.222149 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:13.222075    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" path="/var/lib/kubelet/pods/4db505cc-04e3-406a-8ebc-c75a28754435/volumes"
Sep 16 19:12:24.344953 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.344901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"]
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345209    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345220    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345232    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="llm-d-routing-sidecar"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345238    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="llm-d-routing-sidecar"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345247    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7373cc2c-e7f2-4c9d-8035-582d936725ac" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345253    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7373cc2c-e7f2-4c9d-8035-582d936725ac" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345270    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345321    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7373cc2c-e7f2-4c9d-8035-582d936725ac" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345328    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345334    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db505cc-04e3-406a-8ebc-c75a28754435" containerName="main"
Sep 16 19:12:24.345348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.345341    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e769617e-46c1-425e-8e92-98fa08921c60" containerName="llm-d-routing-sidecar"
Sep 16 19:12:24.348341 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.348314    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.351574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.351553    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:12:24.351574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.351562    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-dx47h\""
Sep 16 19:12:24.351912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.351888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:12:24.352051 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.351960    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-g6d4c\""
Sep 16 19:12:24.352168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.352146    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:12:24.353363 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.353333    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"]
Sep 16 19:12:24.356412 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.356389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.361845 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.361822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"]
Sep 16 19:12:24.382985 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.382960    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"]
Sep 16 19:12:24.400354 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.400486 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.400486 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.400567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.400567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef4c9a1-54f8-4695-b366-219599324408-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.400630 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d207cbb-927a-47b8-948d-20e1289b45fc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.400630 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lr2x\" (UniqueName: \"kubernetes.io/projected/6ef4c9a1-54f8-4695-b366-219599324408-kube-api-access-4lr2x\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.400630 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.400731 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.400731 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.400731 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.400822 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.400730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgbwj\" (UniqueName: \"kubernetes.io/projected/9d207cbb-927a-47b8-948d-20e1289b45fc-kube-api-access-fgbwj\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.501817 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.501785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.501991 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.501827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.501991 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.501972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502089 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502089 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef4c9a1-54f8-4695-b366-219599324408-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502089 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d207cbb-927a-47b8-948d-20e1289b45fc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502247 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lr2x\" (UniqueName: \"kubernetes.io/projected/6ef4c9a1-54f8-4695-b366-219599324408-kube-api-access-4lr2x\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502418 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502418 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502418 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgbwj\" (UniqueName: \"kubernetes.io/projected/9d207cbb-927a-47b8-948d-20e1289b45fc-kube-api-access-fgbwj\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502379    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502729 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.502729 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.502851 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.502827    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.504826 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.504800    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.504969 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.504880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.505082 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.505064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef4c9a1-54f8-4695-b366-219599324408-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.505335 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.505313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d207cbb-927a-47b8-948d-20e1289b45fc-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.510789 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.510771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lr2x\" (UniqueName: \"kubernetes.io/projected/6ef4c9a1-54f8-4695-b366-219599324408-kube-api-access-4lr2x\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.511226 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.511203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgbwj\" (UniqueName: \"kubernetes.io/projected/9d207cbb-927a-47b8-948d-20e1289b45fc-kube-api-access-fgbwj\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.566066 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.566035    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"]
Sep 16 19:12:24.570743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.570723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.574001 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.573978    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-hwckx\""
Sep 16 19:12:24.583064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.583042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"]
Sep 16 19:12:24.603203 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.603151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.603290 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.603213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ccrk\" (UniqueName: \"kubernetes.io/projected/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-kube-api-access-8ccrk\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.659294 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.659267    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:24.668228 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.668208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:24.704672 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.704603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccrk\" (UniqueName: \"kubernetes.io/projected/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-kube-api-access-8ccrk\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.704834 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.704763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.708373 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.708341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.713763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.713730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccrk\" (UniqueName: \"kubernetes.io/projected/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-kube-api-access-8ccrk\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.881447 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.881367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:24.957939 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.957908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"]
Sep 16 19:12:24.959982 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:12:24.959958    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d207cbb_927a_47b8_948d_20e1289b45fc.slice/crio-b1b2d715256eca6e20ca7b3b42d1b9802ea07b65ddc798e4fc07fc97e27ea790 WatchSource:0}: Error finding container b1b2d715256eca6e20ca7b3b42d1b9802ea07b65ddc798e4fc07fc97e27ea790: Status 404 returned error can't find the container with id b1b2d715256eca6e20ca7b3b42d1b9802ea07b65ddc798e4fc07fc97e27ea790
Sep 16 19:12:24.967257 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:24.967237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"]
Sep 16 19:12:24.969851 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:12:24.969824    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef4c9a1_54f8_4695_b366_219599324408.slice/crio-7152d4ca980c798a692f7c0214c7771093cd20c7228ce264911d85abb34dc047 WatchSource:0}: Error finding container 7152d4ca980c798a692f7c0214c7771093cd20c7228ce264911d85abb34dc047: Status 404 returned error can't find the container with id 7152d4ca980c798a692f7c0214c7771093cd20c7228ce264911d85abb34dc047
Sep 16 19:12:25.011914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:25.011886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" event={"ID":"6ef4c9a1-54f8-4695-b366-219599324408","Type":"ContainerStarted","Data":"7152d4ca980c798a692f7c0214c7771093cd20c7228ce264911d85abb34dc047"}
Sep 16 19:12:25.013217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:25.013197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" event={"ID":"9d207cbb-927a-47b8-948d-20e1289b45fc","Type":"ContainerStarted","Data":"b1b2d715256eca6e20ca7b3b42d1b9802ea07b65ddc798e4fc07fc97e27ea790"}
Sep 16 19:12:25.153813 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:25.153787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"]
Sep 16 19:12:25.156193 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:12:25.156166    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9e9bde_14ff_47ff_9e50_b7a4d81226e6.slice/crio-f98a67b9a3cfc581554fd4b7ec7e3f98ec647d684defac03137140ee56daa411 WatchSource:0}: Error finding container f98a67b9a3cfc581554fd4b7ec7e3f98ec647d684defac03137140ee56daa411: Status 404 returned error can't find the container with id f98a67b9a3cfc581554fd4b7ec7e3f98ec647d684defac03137140ee56daa411
Sep 16 19:12:26.020249 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.020209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" event={"ID":"6ef4c9a1-54f8-4695-b366-219599324408","Type":"ContainerStarted","Data":"8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"}
Sep 16 19:12:26.021866 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.021828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" event={"ID":"9d207cbb-927a-47b8-948d-20e1289b45fc","Type":"ContainerStarted","Data":"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"}
Sep 16 19:12:26.022005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.021888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:26.023453 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.023432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" event={"ID":"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6","Type":"ContainerStarted","Data":"47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97"}
Sep 16 19:12:26.023553 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.023466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" event={"ID":"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6","Type":"ContainerStarted","Data":"f98a67b9a3cfc581554fd4b7ec7e3f98ec647d684defac03137140ee56daa411"}
Sep 16 19:12:26.023590 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.023566    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:26.038987 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.038949    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" podStartSLOduration=2.038933376 podStartE2EDuration="2.038933376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:26.036719713 +0000 UTC m=+1647.340736674" watchObservedRunningTime="2026-09-16 19:12:26.038933376 +0000 UTC m=+1647.342950343"
Sep 16 19:12:26.055874 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.055823    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" podStartSLOduration=2.055808605 podStartE2EDuration="2.055808605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:26.054321057 +0000 UTC m=+1647.358338049" watchObservedRunningTime="2026-09-16 19:12:26.055808605 +0000 UTC m=+1647.359825572"
Sep 16 19:12:26.669281 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:26.669247    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:27.030627 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:27.030530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" event={"ID":"9d207cbb-927a-47b8-948d-20e1289b45fc","Type":"ContainerStarted","Data":"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"}
Sep 16 19:12:27.054750 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:27.054701    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" podStartSLOduration=3.054685051 podStartE2EDuration="3.054685051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:27.053517631 +0000 UTC m=+1648.357534598" watchObservedRunningTime="2026-09-16 19:12:27.054685051 +0000 UTC m=+1648.358702017"
Sep 16 19:12:27.660307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:27.660277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:34.659583 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:34.659546    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:34.668910 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:34.668887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:34.672277 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:34.672256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:34.681734 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:34.681713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:35.081517 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:35.081441    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:35.087217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:35.087191    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:35.087352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:35.087250    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:57.033001 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:57.032969    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:12:59.373526 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.373396    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"]
Sep 16 19:12:59.377032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.377003    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"]
Sep 16 19:12:59.381623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.381596    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"]
Sep 16 19:12:59.387769 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:12:59.387734    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:12:59.387769 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:12:59.387769 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:12:59.387769 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc" cmd=["/bin/sleep","15"]
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.387793    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" message="" err=<
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:12:59.387809    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" podUID="6ef4c9a1-54f8-4695-b366-219599324408" containerName="main" containerID="cri-o://8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"
Sep 16 19:12:59.388082 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.387851    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" podUID="6ef4c9a1-54f8-4695-b366-219599324408" containerName="main" containerID="cri-o://8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc" gracePeriod=60
Sep 16 19:12:59.394612 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:12:59.394580    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:12:59.394612 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:12:59.394612 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:12:59.394612 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e" cmd=["/bin/sleep","15"]
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.394631    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" message="" err=<
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:12:59.394647    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="main" containerID="cri-o://b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"
Sep 16 19:12:59.394898 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.394688    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="main" containerID="cri-o://b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e" gracePeriod=60
Sep 16 19:12:59.605725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.605536    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="llm-d-routing-sidecar" containerID="cri-o://6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853" gracePeriod=60
Sep 16 19:12:59.737768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.737744    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:12:59.851143 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.851114    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:12:59.887587 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-model-cache\") pod \"6ef4c9a1-54f8-4695-b366-219599324408\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") "
Sep 16 19:12:59.887733 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lr2x\" (UniqueName: \"kubernetes.io/projected/6ef4c9a1-54f8-4695-b366-219599324408-kube-api-access-4lr2x\") pod \"6ef4c9a1-54f8-4695-b366-219599324408\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") "
Sep 16 19:12:59.887733 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef4c9a1-54f8-4695-b366-219599324408-tls-certs\") pod \"6ef4c9a1-54f8-4695-b366-219599324408\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") "
Sep 16 19:12:59.887733 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-home\") pod \"6ef4c9a1-54f8-4695-b366-219599324408\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") "
Sep 16 19:12:59.887733 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-dshm\") pod \"6ef4c9a1-54f8-4695-b366-219599324408\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") "
Sep 16 19:12:59.887733 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887725    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-tmp-dir\") pod \"6ef4c9a1-54f8-4695-b366-219599324408\" (UID: \"6ef4c9a1-54f8-4695-b366-219599324408\") "
Sep 16 19:12:59.888029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ef4c9a1-54f8-4695-b366-219599324408" (UID: "6ef4c9a1-54f8-4695-b366-219599324408"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.888029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-home" (OuterVolumeSpecName: "home") pod "6ef4c9a1-54f8-4695-b366-219599324408" (UID: "6ef4c9a1-54f8-4695-b366-219599324408"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.888029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.887997    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.888253 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.888226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ef4c9a1-54f8-4695-b366-219599324408" (UID: "6ef4c9a1-54f8-4695-b366-219599324408"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.890008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.889988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-dshm" (OuterVolumeSpecName: "dshm") pod "6ef4c9a1-54f8-4695-b366-219599324408" (UID: "6ef4c9a1-54f8-4695-b366-219599324408"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.890008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.889989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef4c9a1-54f8-4695-b366-219599324408-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ef4c9a1-54f8-4695-b366-219599324408" (UID: "6ef4c9a1-54f8-4695-b366-219599324408"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:59.890130 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.890050    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef4c9a1-54f8-4695-b366-219599324408-kube-api-access-4lr2x" (OuterVolumeSpecName: "kube-api-access-4lr2x") pod "6ef4c9a1-54f8-4695-b366-219599324408" (UID: "6ef4c9a1-54f8-4695-b366-219599324408"). InnerVolumeSpecName "kube-api-access-4lr2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:59.988892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.988804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgbwj\" (UniqueName: \"kubernetes.io/projected/9d207cbb-927a-47b8-948d-20e1289b45fc-kube-api-access-fgbwj\") pod \"9d207cbb-927a-47b8-948d-20e1289b45fc\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") "
Sep 16 19:12:59.988892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.988836    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-tmp-dir\") pod \"9d207cbb-927a-47b8-948d-20e1289b45fc\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") "
Sep 16 19:12:59.989077 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.988910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d207cbb-927a-47b8-948d-20e1289b45fc-tls-certs\") pod \"9d207cbb-927a-47b8-948d-20e1289b45fc\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") "
Sep 16 19:12:59.989077 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.988942    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-dshm\") pod \"9d207cbb-927a-47b8-948d-20e1289b45fc\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") "
Sep 16 19:12:59.989077 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.988963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-home\") pod \"9d207cbb-927a-47b8-948d-20e1289b45fc\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") "
Sep 16 19:12:59.989241 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-model-cache\") pod \"9d207cbb-927a-47b8-948d-20e1289b45fc\" (UID: \"9d207cbb-927a-47b8-948d-20e1289b45fc\") "
Sep 16 19:12:59.989241 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d207cbb-927a-47b8-948d-20e1289b45fc" (UID: "9d207cbb-927a-47b8-948d-20e1289b45fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.989241 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989213    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-home" (OuterVolumeSpecName: "home") pod "9d207cbb-927a-47b8-948d-20e1289b45fc" (UID: "9d207cbb-927a-47b8-948d-20e1289b45fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.989375 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989334    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d207cbb-927a-47b8-948d-20e1289b45fc" (UID: "9d207cbb-927a-47b8-948d-20e1289b45fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.989416 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989407    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.989450 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989425    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.989450 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989441    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.989522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989453    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.989522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989467    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lr2x\" (UniqueName: \"kubernetes.io/projected/6ef4c9a1-54f8-4695-b366-219599324408-kube-api-access-4lr2x\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.989522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989482    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ef4c9a1-54f8-4695-b366-219599324408-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.989522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.989494    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ef4c9a1-54f8-4695-b366-219599324408-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:12:59.990912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.990887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d207cbb-927a-47b8-948d-20e1289b45fc-kube-api-access-fgbwj" (OuterVolumeSpecName: "kube-api-access-fgbwj") pod "9d207cbb-927a-47b8-948d-20e1289b45fc" (UID: "9d207cbb-927a-47b8-948d-20e1289b45fc"). InnerVolumeSpecName "kube-api-access-fgbwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:59.990912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.990888    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-dshm" (OuterVolumeSpecName: "dshm") pod "9d207cbb-927a-47b8-948d-20e1289b45fc" (UID: "9d207cbb-927a-47b8-948d-20e1289b45fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:59.991093 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:12:59.991073    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d207cbb-927a-47b8-948d-20e1289b45fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d207cbb-927a-47b8-948d-20e1289b45fc" (UID: "9d207cbb-927a-47b8-948d-20e1289b45fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:00.089836 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.089802    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d207cbb-927a-47b8-948d-20e1289b45fc-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:00.089836 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.089827    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:00.089836 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.089836    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d207cbb-927a-47b8-948d-20e1289b45fc-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:00.089836 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.089844    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgbwj\" (UniqueName: \"kubernetes.io/projected/9d207cbb-927a-47b8-948d-20e1289b45fc-kube-api-access-fgbwj\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:00.195055 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.195015    2581 generic.go:356] "Generic (PLEG): container finished" podID="6ef4c9a1-54f8-4695-b366-219599324408" containerID="8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc" exitCode=0
Sep 16 19:13:00.195231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.195064    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"
Sep 16 19:13:00.195231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.195096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" event={"ID":"6ef4c9a1-54f8-4695-b366-219599324408","Type":"ContainerDied","Data":"8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"}
Sep 16 19:13:00.195231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.195175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2" event={"ID":"6ef4c9a1-54f8-4695-b366-219599324408","Type":"ContainerDied","Data":"7152d4ca980c798a692f7c0214c7771093cd20c7228ce264911d85abb34dc047"}
Sep 16 19:13:00.195231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.195207    2581 scope.go:117] "RemoveContainer" containerID="8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"
Sep 16 19:13:00.196761 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.196735    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerID="b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e" exitCode=0
Sep 16 19:13:00.196761 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.196759    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerID="6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853" exitCode=0
Sep 16 19:13:00.196962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.196785    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"
Sep 16 19:13:00.196962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.196787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" event={"ID":"9d207cbb-927a-47b8-948d-20e1289b45fc","Type":"ContainerDied","Data":"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"}
Sep 16 19:13:00.196962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.196905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" event={"ID":"9d207cbb-927a-47b8-948d-20e1289b45fc","Type":"ContainerDied","Data":"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"}
Sep 16 19:13:00.196962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.196930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh" event={"ID":"9d207cbb-927a-47b8-948d-20e1289b45fc","Type":"ContainerDied","Data":"b1b2d715256eca6e20ca7b3b42d1b9802ea07b65ddc798e4fc07fc97e27ea790"}
Sep 16 19:13:00.205141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.205124    2581 scope.go:117] "RemoveContainer" containerID="8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"
Sep 16 19:13:00.205436 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:13:00.205419    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc\": container with ID starting with 8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc not found: ID does not exist" containerID="8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"
Sep 16 19:13:00.205524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.205447    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc"} err="failed to get container status \"8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc\": rpc error: code = NotFound desc = could not find container \"8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc\": container with ID starting with 8a78b9e47c09d959d47375d986a0dea7c65db93f7b7eceefdcac5e2be32d50dc not found: ID does not exist"
Sep 16 19:13:00.205524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.205468    2581 scope.go:117] "RemoveContainer" containerID="b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"
Sep 16 19:13:00.213208 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.213191    2581 scope.go:117] "RemoveContainer" containerID="6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"
Sep 16 19:13:00.220991 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.220909    2581 scope.go:117] "RemoveContainer" containerID="b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"
Sep 16 19:13:00.221193 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:13:00.221173    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e\": container with ID starting with b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e not found: ID does not exist" containerID="b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"
Sep 16 19:13:00.221256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221199    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"} err="failed to get container status \"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e\": rpc error: code = NotFound desc = could not find container \"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e\": container with ID starting with b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e not found: ID does not exist"
Sep 16 19:13:00.221256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221215    2581 scope.go:117] "RemoveContainer" containerID="6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"
Sep 16 19:13:00.221359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221279    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"]
Sep 16 19:13:00.221463 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:13:00.221444    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853\": container with ID starting with 6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853 not found: ID does not exist" containerID="6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"
Sep 16 19:13:00.221504 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221473    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"} err="failed to get container status \"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853\": rpc error: code = NotFound desc = could not find container \"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853\": container with ID starting with 6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853 not found: ID does not exist"
Sep 16 19:13:00.221504 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221489    2581 scope.go:117] "RemoveContainer" containerID="b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"
Sep 16 19:13:00.221731 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221712    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e"} err="failed to get container status \"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e\": rpc error: code = NotFound desc = could not find container \"b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e\": container with ID starting with b456404c9c8636de504929df2e55ce3d6aa307de6d07e8f2fee47b2bae0fb07e not found: ID does not exist"
Sep 16 19:13:00.221803 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221733    2581 scope.go:117] "RemoveContainer" containerID="6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"
Sep 16 19:13:00.221981 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.221963    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853"} err="failed to get container status \"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853\": rpc error: code = NotFound desc = could not find container \"6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853\": container with ID starting with 6b10edb7ee42b54304b9c57f337fef9d31e9fa6556a244e9679253df8c6b0853 not found: ID does not exist"
Sep 16 19:13:00.226895 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.226875    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-8w7dh"]
Sep 16 19:13:00.239596 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.239543    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"]
Sep 16 19:13:00.243913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:00.243894    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444sp4x2"]
Sep 16 19:13:01.221020 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:01.220990    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ef4c9a1-54f8-4695-b366-219599324408" path="/var/lib/kubelet/pods/6ef4c9a1-54f8-4695-b366-219599324408/volumes"
Sep 16 19:13:01.221410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:01.221395    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" path="/var/lib/kubelet/pods/9d207cbb-927a-47b8-948d-20e1289b45fc/volumes"
Sep 16 19:13:07.032501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:07.032465    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:14.395534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:14.395496    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main" containerID="cri-o://47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97" gracePeriod=45
Sep 16 19:13:17.032752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:17.032707    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:27.032387 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:27.032350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:27.032787 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:27.032459    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:13:37.032838 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:37.032799    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:13:44.639994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.639961    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:13:44.729344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.729277    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ccrk\" (UniqueName: \"kubernetes.io/projected/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-kube-api-access-8ccrk\") pod \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") "
Sep 16 19:13:44.729344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.729315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-tls-certs\") pod \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\" (UID: \"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6\") "
Sep 16 19:13:44.731332 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.731302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" (UID: "2d9e9bde-14ff-47ff-9e50-b7a4d81226e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:44.731427 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.731301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-kube-api-access-8ccrk" (OuterVolumeSpecName: "kube-api-access-8ccrk") pod "2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" (UID: "2d9e9bde-14ff-47ff-9e50-b7a4d81226e6"). InnerVolumeSpecName "kube-api-access-8ccrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:44.829939 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.829911    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ccrk\" (UniqueName: \"kubernetes.io/projected/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-kube-api-access-8ccrk\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:44.829939 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:44.829934    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:13:45.423244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.423207    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerID="47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97" exitCode=0
Sep 16 19:13:45.423417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.423277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"
Sep 16 19:13:45.423417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.423289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" event={"ID":"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6","Type":"ContainerDied","Data":"47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97"}
Sep 16 19:13:45.423417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.423354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9" event={"ID":"2d9e9bde-14ff-47ff-9e50-b7a4d81226e6","Type":"ContainerDied","Data":"f98a67b9a3cfc581554fd4b7ec7e3f98ec647d684defac03137140ee56daa411"}
Sep 16 19:13:45.424141 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.423382    2581 scope.go:117] "RemoveContainer" containerID="47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97"
Sep 16 19:13:45.435680 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.435659    2581 scope.go:117] "RemoveContainer" containerID="47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97"
Sep 16 19:13:45.436166 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:13:45.436143    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97\": container with ID starting with 47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97 not found: ID does not exist" containerID="47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97"
Sep 16 19:13:45.436250 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.436178    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97"} err="failed to get container status \"47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97\": rpc error: code = NotFound desc = could not find container \"47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97\": container with ID starting with 47b70394d9f6e6eb604bcac5cf2eb1f7725e7826c18df15c5936e1415b514e97 not found: ID does not exist"
Sep 16 19:13:45.441617 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.441596    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"]
Sep 16 19:13:45.448906 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:45.448885    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedf5lk9"]
Sep 16 19:13:46.311318 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311286    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"]
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311579    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="llm-d-routing-sidecar"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311590    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="llm-d-routing-sidecar"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311604    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311609    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311619    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ef4c9a1-54f8-4695-b366-219599324408" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311625    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ef4c9a1-54f8-4695-b366-219599324408" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311646    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311699    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311708    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ef4c9a1-54f8-4695-b366-219599324408" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311713    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="main"
Sep 16 19:13:46.311779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.311720    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d207cbb-927a-47b8-948d-20e1289b45fc" containerName="llm-d-routing-sidecar"
Sep 16 19:13:46.315596 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.315576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.320101 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.320077    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:13:46.320101 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.320099    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:13:46.320352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.320200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-zprnr\""
Sep 16 19:13:46.320455 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.320436    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 19:13:46.327412 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.327389    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"]
Sep 16 19:13:46.441405 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.441374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f550b87d-17b4-4afd-84d8-18355ebb325e-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.441405 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.441411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-home\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.441648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.441439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-dshm\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.441648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.441509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzkh\" (UniqueName: \"kubernetes.io/projected/f550b87d-17b4-4afd-84d8-18355ebb325e-kube-api-access-nqzkh\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.441648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.441553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-model-cache\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.441648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.441582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f550b87d-17b4-4afd-84d8-18355ebb325e-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-home\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-dshm\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzkh\" (UniqueName: \"kubernetes.io/projected/f550b87d-17b4-4afd-84d8-18355ebb325e-kube-api-access-nqzkh\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-model-cache\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542551 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542551 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-home\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.542784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.542723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-model-cache\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.544618 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.544598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f550b87d-17b4-4afd-84d8-18355ebb325e-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.544757 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.544733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-dshm\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.551495 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.551473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzkh\" (UniqueName: \"kubernetes.io/projected/f550b87d-17b4-4afd-84d8-18355ebb325e-kube-api-access-nqzkh\") pod \"canary-v1-kserve-65d58ccc9-pj8br\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.626793 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.626714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:46.898156 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:46.898131    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"]
Sep 16 19:13:46.900686 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:13:46.900647    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf550b87d_17b4_4afd_84d8_18355ebb325e.slice/crio-923fd2d07fdb0e05255b11d061ae18ba780a4038c0c3c53eaf965974740dad3c WatchSource:0}: Error finding container 923fd2d07fdb0e05255b11d061ae18ba780a4038c0c3c53eaf965974740dad3c: Status 404 returned error can't find the container with id 923fd2d07fdb0e05255b11d061ae18ba780a4038c0c3c53eaf965974740dad3c
Sep 16 19:13:47.222669 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:47.222614    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9e9bde-14ff-47ff-9e50-b7a4d81226e6" path="/var/lib/kubelet/pods/2d9e9bde-14ff-47ff-9e50-b7a4d81226e6/volumes"
Sep 16 19:13:47.438060 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:47.438019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br" event={"ID":"f550b87d-17b4-4afd-84d8-18355ebb325e","Type":"ContainerStarted","Data":"923fd2d07fdb0e05255b11d061ae18ba780a4038c0c3c53eaf965974740dad3c"}
Sep 16 19:13:48.447957 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:48.447145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br" event={"ID":"f550b87d-17b4-4afd-84d8-18355ebb325e","Type":"ContainerStarted","Data":"75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f"}
Sep 16 19:13:48.471372 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:48.471326    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br" podStartSLOduration=1.18185035 podStartE2EDuration="2.471310007s" totalImagesPullingTime="1.289459657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:48.468299719 +0000 UTC m=+1729.772316698" watchObservedRunningTime="2026-09-16 19:13:48.471310007 +0000 UTC m=+1729.775326973"
Sep 16 19:13:48.627262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:48.627233    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:56.627205 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:56.627166    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:56.639637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:56.639609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:13:57.506771 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:13:57.506739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:14:54.341985 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.341912    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"]
Sep 16 19:14:54.344329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.344307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.347492 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.347471    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:14:54.348589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.348568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-k2pfb\""
Sep 16 19:14:54.348715 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.348587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 19:14:54.348715 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.348602    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:14:54.348715 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.348618    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:14:54.348715 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.348641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:14:54.361539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.361512    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"]
Sep 16 19:14:54.508353 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508353 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-home\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17c9f23e-1018-4b41-bca2-f76cdd707a4b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rkd2\" (UniqueName: \"kubernetes.io/projected/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kube-api-access-2rkd2\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-dshm\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.508591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.508571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.589954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.589921    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"]
Sep 16 19:14:54.592110 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.592067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.609264 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-home\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17c9f23e-1018-4b41-bca2-f76cdd707a4b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rkd2\" (UniqueName: \"kubernetes.io/projected/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kube-api-access-2rkd2\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-dshm\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609490    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-home\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.609989 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.609959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.610084 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.610004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-model-cache\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.610144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.610106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.610144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.610124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17c9f23e-1018-4b41-bca2-f76cdd707a4b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.610586 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.610563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"]
Sep 16 19:14:54.612142 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.612117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-dshm\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.612247 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.612182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.631159 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.631136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rkd2\" (UniqueName: \"kubernetes.io/projected/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kube-api-access-2rkd2\") pod \"tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.654278 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.654252    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:14:54.710470 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.710438    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e93cef-3add-4b01-8a36-a6e593a646dc-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.710600 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.710483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.710600 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.710576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e93cef-3add-4b01-8a36-a6e593a646dc-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.710717 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.710671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.811395 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.811358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e93cef-3add-4b01-8a36-a6e593a646dc-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.811576 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.811423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.811576 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.811454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e93cef-3add-4b01-8a36-a6e593a646dc-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.811576 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.811499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.811878 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.811836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.811951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.811882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.812159 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.812140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e93cef-3add-4b01-8a36-a6e593a646dc-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.813948 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.813929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e93cef-3add-4b01-8a36-a6e593a646dc-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.902082 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.902049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:14:54.932950 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:54.932921    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"]
Sep 16 19:14:54.934243 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:14:54.934220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c9f23e_1018_4b41_bca2_f76cdd707a4b.slice/crio-abbffef9b0ff87ba6fe4b9d31788caa866acc1cd68b4ec448e42d87119d34a11 WatchSource:0}: Error finding container abbffef9b0ff87ba6fe4b9d31788caa866acc1cd68b4ec448e42d87119d34a11: Status 404 returned error can't find the container with id abbffef9b0ff87ba6fe4b9d31788caa866acc1cd68b4ec448e42d87119d34a11
Sep 16 19:14:55.184099 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:55.184021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"]
Sep 16 19:14:55.188046 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:14:55.188018    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e93cef_3add_4b01_8a36_a6e593a646dc.slice/crio-a69273bc9fc64eb9da06f99fa0e5883b667cef5e3d74229995dc615192698227 WatchSource:0}: Error finding container a69273bc9fc64eb9da06f99fa0e5883b667cef5e3d74229995dc615192698227: Status 404 returned error can't find the container with id a69273bc9fc64eb9da06f99fa0e5883b667cef5e3d74229995dc615192698227
Sep 16 19:14:55.783895 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:55.783835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" event={"ID":"13e93cef-3add-4b01-8a36-a6e593a646dc","Type":"ContainerStarted","Data":"3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4"}
Sep 16 19:14:55.784409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:55.783920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" event={"ID":"13e93cef-3add-4b01-8a36-a6e593a646dc","Type":"ContainerStarted","Data":"a69273bc9fc64eb9da06f99fa0e5883b667cef5e3d74229995dc615192698227"}
Sep 16 19:14:55.785777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:55.785751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" event={"ID":"17c9f23e-1018-4b41-bca2-f76cdd707a4b","Type":"ContainerStarted","Data":"de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75"}
Sep 16 19:14:55.785948 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:55.785797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" event={"ID":"17c9f23e-1018-4b41-bca2-f76cdd707a4b","Type":"ContainerStarted","Data":"abbffef9b0ff87ba6fe4b9d31788caa866acc1cd68b4ec448e42d87119d34a11"}
Sep 16 19:14:57.800958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:57.800923    2581 generic.go:356] "Generic (PLEG): container finished" podID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerID="de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75" exitCode=0
Sep 16 19:14:57.801463 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:57.800993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" event={"ID":"17c9f23e-1018-4b41-bca2-f76cdd707a4b","Type":"ContainerDied","Data":"de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75"}
Sep 16 19:14:58.810648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:58.810614    2581 generic.go:356] "Generic (PLEG): container finished" podID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerID="3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4" exitCode=0
Sep 16 19:14:58.811102 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:58.810692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" event={"ID":"13e93cef-3add-4b01-8a36-a6e593a646dc","Type":"ContainerDied","Data":"3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4"}
Sep 16 19:14:58.812754 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:58.812730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" event={"ID":"17c9f23e-1018-4b41-bca2-f76cdd707a4b","Type":"ContainerStarted","Data":"f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"}
Sep 16 19:14:58.867031 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:58.866977    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" podStartSLOduration=2.220736254 podStartE2EDuration="4.866963628s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.646227374s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:58.864948865 +0000 UTC m=+1800.168965832" watchObservedRunningTime="2026-09-16 19:14:58.866963628 +0000 UTC m=+1800.170980594"
Sep 16 19:14:59.297821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:59.297791    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:14:59.298343 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:59.298327    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:14:59.300875 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:59.300839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:14:59.301261 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:59.301248    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:14:59.654344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:14:59.654315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:15:04.655307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:15:04.655265    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:15:04.674149 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:15:04.674117    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:15:04.865356 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:15:04.865324    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:15:48.119092 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:15:48.119057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" event={"ID":"13e93cef-3add-4b01-8a36-a6e593a646dc","Type":"ContainerStarted","Data":"25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e"}
Sep 16 19:15:48.119587 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:15:48.119268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:15:48.146990 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:15:48.143049    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" podStartSLOduration=3.069220881 podStartE2EDuration="54.143027044s" totalImagesPullingTime="48.401051125s" totalInitContainerRuntime="2.672755038s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:48.138818784 +0000 UTC m=+1849.442835754" watchObservedRunningTime="2026-09-16 19:15:48.143027044 +0000 UTC m=+1849.447044012"
Sep 16 19:16:19.136245 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:19.136212    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:16:21.155098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.155063    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk_13e93cef-3add-4b01-8a36-a6e593a646dc/main/0.log"
Sep 16 19:16:21.544524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.544441    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"]
Sep 16 19:16:21.544688 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.544648    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerName="main" containerID="cri-o://25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e" gracePeriod=30
Sep 16 19:16:21.552473 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.552446    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"]
Sep 16 19:16:21.563020 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:16:21.562982    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:16:21.563020 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:16:21.563020 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:16:21.563020 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df" cmd=["/bin/sleep","15"]
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.563045    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" message="" err=<
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:16:21.563064    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="main" containerID="cri-o://f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"
Sep 16 19:16:21.563342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.563112    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="main" containerID="cri-o://f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df" gracePeriod=60
Sep 16 19:16:21.959732 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:21.959694    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:16:22.091773 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-dshm\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.091934 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091788    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-model-cache\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.091934 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091816    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tmp-dir\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.091934 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17c9f23e-1018-4b41-bca2-f76cdd707a4b-cabundle-cert\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.091934 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091905    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rkd2\" (UniqueName: \"kubernetes.io/projected/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kube-api-access-2rkd2\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.092175 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tls-certs\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.092175 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.091981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-home\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.092175 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kserve-provision-location\") pod \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\" (UID: \"17c9f23e-1018-4b41-bca2-f76cdd707a4b\") "
Sep 16 19:16:22.092175 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092052    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.092175 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.092424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092191    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c9f23e-1018-4b41-bca2-f76cdd707a4b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:22.092424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092358    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.092424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092383    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.092424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092396    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17c9f23e-1018-4b41-bca2-f76cdd707a4b-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.092424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.092358    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-home" (OuterVolumeSpecName: "home") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.094150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.094127    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:22.094352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.094332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kube-api-access-2rkd2" (OuterVolumeSpecName: "kube-api-access-2rkd2") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "kube-api-access-2rkd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:22.094837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.094820    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-dshm" (OuterVolumeSpecName: "dshm") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.113853 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.113825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17c9f23e-1018-4b41-bca2-f76cdd707a4b" (UID: "17c9f23e-1018-4b41-bca2-f76cdd707a4b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.193103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.193080    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.193103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.193104    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rkd2\" (UniqueName: \"kubernetes.io/projected/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kube-api-access-2rkd2\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.193505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.193115    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17c9f23e-1018-4b41-bca2-f76cdd707a4b-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.193505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.193125    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.193505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.193134    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17c9f23e-1018-4b41-bca2-f76cdd707a4b-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.224168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.224142    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:16:22.293334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293296    2581 generic.go:356] "Generic (PLEG): container finished" podID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerID="25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e" exitCode=0
Sep 16 19:16:22.293507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"
Sep 16 19:16:22.293507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293362    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e93cef-3add-4b01-8a36-a6e593a646dc-tls-certs\") pod \"13e93cef-3add-4b01-8a36-a6e593a646dc\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") "
Sep 16 19:16:22.293507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293416    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-kserve-provision-location\") pod \"13e93cef-3add-4b01-8a36-a6e593a646dc\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") "
Sep 16 19:16:22.293507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293466    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-tmp-dir\") pod \"13e93cef-3add-4b01-8a36-a6e593a646dc\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") "
Sep 16 19:16:22.293507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" event={"ID":"13e93cef-3add-4b01-8a36-a6e593a646dc","Type":"ContainerDied","Data":"25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e"}
Sep 16 19:16:22.293507 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e93cef-3add-4b01-8a36-a6e593a646dc-cabundle-cert\") pod \"13e93cef-3add-4b01-8a36-a6e593a646dc\" (UID: \"13e93cef-3add-4b01-8a36-a6e593a646dc\") "
Sep 16 19:16:22.293821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk" event={"ID":"13e93cef-3add-4b01-8a36-a6e593a646dc","Type":"ContainerDied","Data":"a69273bc9fc64eb9da06f99fa0e5883b667cef5e3d74229995dc615192698227"}
Sep 16 19:16:22.293821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.293603    2581 scope.go:117] "RemoveContainer" containerID="25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e"
Sep 16 19:16:22.294038 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.294013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e93cef-3add-4b01-8a36-a6e593a646dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13e93cef-3add-4b01-8a36-a6e593a646dc" (UID: "13e93cef-3add-4b01-8a36-a6e593a646dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:22.294246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.294221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "13e93cef-3add-4b01-8a36-a6e593a646dc" (UID: "13e93cef-3add-4b01-8a36-a6e593a646dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.295425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.295396    2581 generic.go:356] "Generic (PLEG): container finished" podID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerID="f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df" exitCode=0
Sep 16 19:16:22.295542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.295448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" event={"ID":"17c9f23e-1018-4b41-bca2-f76cdd707a4b","Type":"ContainerDied","Data":"f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"}
Sep 16 19:16:22.295542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.295455    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"
Sep 16 19:16:22.295542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.295479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" event={"ID":"17c9f23e-1018-4b41-bca2-f76cdd707a4b","Type":"ContainerDied","Data":"abbffef9b0ff87ba6fe4b9d31788caa866acc1cd68b4ec448e42d87119d34a11"}
Sep 16 19:16:22.295766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.295744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13e93cef-3add-4b01-8a36-a6e593a646dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "13e93cef-3add-4b01-8a36-a6e593a646dc" (UID: "13e93cef-3add-4b01-8a36-a6e593a646dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:22.302833 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.302814    2581 scope.go:117] "RemoveContainer" containerID="3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4"
Sep 16 19:16:22.312091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.312071    2581 scope.go:117] "RemoveContainer" containerID="25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e"
Sep 16 19:16:22.312340 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:16:22.312322    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e\": container with ID starting with 25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e not found: ID does not exist" containerID="25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e"
Sep 16 19:16:22.312397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.312347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e"} err="failed to get container status \"25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e\": rpc error: code = NotFound desc = could not find container \"25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e\": container with ID starting with 25d3e62561828c2a1241ae8d3ca53fa9cd27303148c6af469ff5e72a6589683e not found: ID does not exist"
Sep 16 19:16:22.312397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.312368    2581 scope.go:117] "RemoveContainer" containerID="3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4"
Sep 16 19:16:22.312645 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:16:22.312625    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4\": container with ID starting with 3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4 not found: ID does not exist" containerID="3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4"
Sep 16 19:16:22.312696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.312648    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4"} err="failed to get container status \"3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4\": rpc error: code = NotFound desc = could not find container \"3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4\": container with ID starting with 3a67d90f871d9955f672e12293626383f20bf8bd566261bfc71f752a6f9d8df4 not found: ID does not exist"
Sep 16 19:16:22.312696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.312662    2581 scope.go:117] "RemoveContainer" containerID="f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"
Sep 16 19:16:22.317589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.317563    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13e93cef-3add-4b01-8a36-a6e593a646dc" (UID: "13e93cef-3add-4b01-8a36-a6e593a646dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:22.320600 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.320581    2581 scope.go:117] "RemoveContainer" containerID="de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75"
Sep 16 19:16:22.322380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.322357    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"]
Sep 16 19:16:22.328144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.328126    2581 scope.go:117] "RemoveContainer" containerID="f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"
Sep 16 19:16:22.328410 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:16:22.328391    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df\": container with ID starting with f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df not found: ID does not exist" containerID="f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"
Sep 16 19:16:22.328481 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.328416    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df"} err="failed to get container status \"f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df\": rpc error: code = NotFound desc = could not find container \"f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df\": container with ID starting with f7298b68877c3c0a07bbb220ba21155c5d5b6f831534f4f7d75af915fcc948df not found: ID does not exist"
Sep 16 19:16:22.328481 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.328437    2581 scope.go:117] "RemoveContainer" containerID="de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75"
Sep 16 19:16:22.328691 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:16:22.328671    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75\": container with ID starting with de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75 not found: ID does not exist" containerID="de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75"
Sep 16 19:16:22.328746 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.328694    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75"} err="failed to get container status \"de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75\": rpc error: code = NotFound desc = could not find container \"de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75\": container with ID starting with de0bf5c6def35e92636957349c4624a3f51a696a4ed2c6316dc918c9d5b29d75 not found: ID does not exist"
Sep 16 19:16:22.329593 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.329570    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk"]
Sep 16 19:16:22.394926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.394841    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13e93cef-3add-4b01-8a36-a6e593a646dc-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.394926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.394887    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.394926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.394901    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13e93cef-3add-4b01-8a36-a6e593a646dc-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.394926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.394915    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13e93cef-3add-4b01-8a36-a6e593a646dc-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:16:22.617719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.617677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"]
Sep 16 19:16:22.621284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.621260    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-69c54d954b-9pfdk"]
Sep 16 19:16:22.852759 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:22.852722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-6b7b9b4855-pj7hk" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:16:23.222054 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:23.222015    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" path="/var/lib/kubelet/pods/13e93cef-3add-4b01-8a36-a6e593a646dc/volumes"
Sep 16 19:16:23.222655 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:16:23.222634    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" path="/var/lib/kubelet/pods/17c9f23e-1018-4b41-bca2-f76cdd707a4b/volumes"
Sep 16 19:17:05.816897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:05.816848    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"]
Sep 16 19:17:20.835660 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:20.835616    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br" podUID="f550b87d-17b4-4afd-84d8-18355ebb325e" containerName="main" containerID="cri-o://75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f" gracePeriod=45
Sep 16 19:17:21.082559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.082535    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:17:21.160169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160144    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-model-cache\") pod \"f550b87d-17b4-4afd-84d8-18355ebb325e\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") "
Sep 16 19:17:21.160363 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160215    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-tmp-dir\") pod \"f550b87d-17b4-4afd-84d8-18355ebb325e\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") "
Sep 16 19:17:21.160363 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqzkh\" (UniqueName: \"kubernetes.io/projected/f550b87d-17b4-4afd-84d8-18355ebb325e-kube-api-access-nqzkh\") pod \"f550b87d-17b4-4afd-84d8-18355ebb325e\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") "
Sep 16 19:17:21.160363 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f550b87d-17b4-4afd-84d8-18355ebb325e-tls-certs\") pod \"f550b87d-17b4-4afd-84d8-18355ebb325e\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") "
Sep 16 19:17:21.160530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-dshm\") pod \"f550b87d-17b4-4afd-84d8-18355ebb325e\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") "
Sep 16 19:17:21.160530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f550b87d-17b4-4afd-84d8-18355ebb325e" (UID: "f550b87d-17b4-4afd-84d8-18355ebb325e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:21.160530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f550b87d-17b4-4afd-84d8-18355ebb325e" (UID: "f550b87d-17b4-4afd-84d8-18355ebb325e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:21.160683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-home\") pod \"f550b87d-17b4-4afd-84d8-18355ebb325e\" (UID: \"f550b87d-17b4-4afd-84d8-18355ebb325e\") "
Sep 16 19:17:21.160804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-home" (OuterVolumeSpecName: "home") pod "f550b87d-17b4-4afd-84d8-18355ebb325e" (UID: "f550b87d-17b4-4afd-84d8-18355ebb325e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:21.160804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160795    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.160942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.160815    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.162497 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.162478    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f550b87d-17b4-4afd-84d8-18355ebb325e-kube-api-access-nqzkh" (OuterVolumeSpecName: "kube-api-access-nqzkh") pod "f550b87d-17b4-4afd-84d8-18355ebb325e" (UID: "f550b87d-17b4-4afd-84d8-18355ebb325e"). InnerVolumeSpecName "kube-api-access-nqzkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:21.162766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.162748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-dshm" (OuterVolumeSpecName: "dshm") pod "f550b87d-17b4-4afd-84d8-18355ebb325e" (UID: "f550b87d-17b4-4afd-84d8-18355ebb325e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:21.162829 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.162766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f550b87d-17b4-4afd-84d8-18355ebb325e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f550b87d-17b4-4afd-84d8-18355ebb325e" (UID: "f550b87d-17b4-4afd-84d8-18355ebb325e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:21.261813 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.261780    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqzkh\" (UniqueName: \"kubernetes.io/projected/f550b87d-17b4-4afd-84d8-18355ebb325e-kube-api-access-nqzkh\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.261813 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.261811    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f550b87d-17b4-4afd-84d8-18355ebb325e-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.262016 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.261824    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.262016 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.261832    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f550b87d-17b4-4afd-84d8-18355ebb325e-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:17:21.599962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.599844    2581 generic.go:356] "Generic (PLEG): container finished" podID="f550b87d-17b4-4afd-84d8-18355ebb325e" containerID="75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f" exitCode=0
Sep 16 19:17:21.599962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.599920    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"
Sep 16 19:17:21.600144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.599921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br" event={"ID":"f550b87d-17b4-4afd-84d8-18355ebb325e","Type":"ContainerDied","Data":"75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f"}
Sep 16 19:17:21.600144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.600054    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br" event={"ID":"f550b87d-17b4-4afd-84d8-18355ebb325e","Type":"ContainerDied","Data":"923fd2d07fdb0e05255b11d061ae18ba780a4038c0c3c53eaf965974740dad3c"}
Sep 16 19:17:21.600144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.600083    2581 scope.go:117] "RemoveContainer" containerID="75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f"
Sep 16 19:17:21.609915 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.609897    2581 scope.go:117] "RemoveContainer" containerID="75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f"
Sep 16 19:17:21.610181 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:17:21.610162    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f\": container with ID starting with 75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f not found: ID does not exist" containerID="75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f"
Sep 16 19:17:21.610252 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.610188    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f"} err="failed to get container status \"75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f\": rpc error: code = NotFound desc = could not find container \"75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f\": container with ID starting with 75bb53182a385b2771bc61ce0f8f977199a37355a149e0f2cbf60f77a743224f not found: ID does not exist"
Sep 16 19:17:21.618265 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.618241    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"]
Sep 16 19:17:21.624164 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:21.624142    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-pj8br"]
Sep 16 19:17:23.222985 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:23.222943    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f550b87d-17b4-4afd-84d8-18355ebb325e" path="/var/lib/kubelet/pods/f550b87d-17b4-4afd-84d8-18355ebb325e/volumes"
Sep 16 19:17:35.012210 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012176    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"]
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012483    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerName="storage-initializer"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerName="storage-initializer"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012504    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012510    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012519    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f550b87d-17b4-4afd-84d8-18355ebb325e" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012524    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f550b87d-17b4-4afd-84d8-18355ebb325e" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012530    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="storage-initializer"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012535    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="storage-initializer"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012550    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012600    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="13e93cef-3add-4b01-8a36-a6e593a646dc" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012607    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="17c9f23e-1018-4b41-bca2-f76cdd707a4b" containerName="main"
Sep 16 19:17:35.012661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.012612    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f550b87d-17b4-4afd-84d8-18355ebb325e" containerName="main"
Sep 16 19:17:35.014547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.014531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.019460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.019437    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:17:35.019734 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.019710    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:17:35.021188 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.020900    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-skkvb\""
Sep 16 19:17:35.021188 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.020984    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:17:35.045415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.045394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"]
Sep 16 19:17:35.072562 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.072540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.072696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.072589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-home\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.072696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.072607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.072696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.072649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.072696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.072665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh2kw\" (UniqueName: \"kubernetes.io/projected/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-kube-api-access-kh2kw\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.072696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.072691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.173898 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.173844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-home\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.173898 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.173895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174114 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174114 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh2kw\" (UniqueName: \"kubernetes.io/projected/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-kube-api-access-kh2kw\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174114 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-home\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174367 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.174453 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.174431    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.176329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.176307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.176575 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.176561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.188094 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.188075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh2kw\" (UniqueName: \"kubernetes.io/projected/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-kube-api-access-kh2kw\") pod \"test-div-v2-kserve-5f6fdb6b77-7mbh6\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.324954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.324890    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:35.597202 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.597178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"]
Sep 16 19:17:35.599550 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:17:35.599515    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee95ac1f_ab40_4d0b_b3c4_79292b997c92.slice/crio-3c42f62bedda1673d3caacd407f307d05353db9a2af63dc293772d98c436979d WatchSource:0}: Error finding container 3c42f62bedda1673d3caacd407f307d05353db9a2af63dc293772d98c436979d: Status 404 returned error can't find the container with id 3c42f62bedda1673d3caacd407f307d05353db9a2af63dc293772d98c436979d
Sep 16 19:17:35.669949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.669918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6" event={"ID":"ee95ac1f-ab40-4d0b-b3c4-79292b997c92","Type":"ContainerStarted","Data":"fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7"}
Sep 16 19:17:35.670072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.669964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6" event={"ID":"ee95ac1f-ab40-4d0b-b3c4-79292b997c92","Type":"ContainerStarted","Data":"3c42f62bedda1673d3caacd407f307d05353db9a2af63dc293772d98c436979d"}
Sep 16 19:17:35.688917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:35.688870    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6" podStartSLOduration=1.6888406200000001 podStartE2EDuration="1.68884062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:35.687194385 +0000 UTC m=+1956.991211352" watchObservedRunningTime="2026-09-16 19:17:35.68884062 +0000 UTC m=+1956.992857587"
Sep 16 19:17:36.325831 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:36.325748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:42.215406 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.215371    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"]
Sep 16 19:17:42.218018 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.217994    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.221829 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.221806    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:17:42.221986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.221900    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 19:17:42.221986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.221814    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:17:42.221986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.221807    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:17:42.221986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.221962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-z7g2v\""
Sep 16 19:17:42.222270 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.221905    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:17:42.232999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.232980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"]
Sep 16 19:17:42.335367 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335335    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-home\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740b3022-61d9-433c-9c51-7e9acb8345a2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/740b3022-61d9-433c-9c51-7e9acb8345a2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335524 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335667 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-dshm\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335667 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-model-cache\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.335667 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.335610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7rxt\" (UniqueName: \"kubernetes.io/projected/740b3022-61d9-433c-9c51-7e9acb8345a2-kube-api-access-k7rxt\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740b3022-61d9-433c-9c51-7e9acb8345a2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/740b3022-61d9-433c-9c51-7e9acb8345a2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-dshm\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-model-cache\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7rxt\" (UniqueName: \"kubernetes.io/projected/740b3022-61d9-433c-9c51-7e9acb8345a2-kube-api-access-k7rxt\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.436763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.436725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-home\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.437183 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.437146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.437265 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.437198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.437324 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.437288    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-home\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.437324 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.437290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-model-cache\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.437476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.437454    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/740b3022-61d9-433c-9c51-7e9acb8345a2-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.439150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.439126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-dshm\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.439405 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.439389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740b3022-61d9-433c-9c51-7e9acb8345a2-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.447529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.447507    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7rxt\" (UniqueName: \"kubernetes.io/projected/740b3022-61d9-433c-9c51-7e9acb8345a2-kube-api-access-k7rxt\") pod \"tokenizer-migration-path-test-kserve-56579b9954-tkbl5\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.472436 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.472378    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"]
Sep 16 19:17:42.475460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.475439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.486953 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.486931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"]
Sep 16 19:17:42.528409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.528379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:42.537525 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.537493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.537642 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.537618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.537714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.537694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.537795 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.537776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.638678 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.638644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.638678 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.638688    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.638919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.638727    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.638919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.638749    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.639173 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.639138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.639230 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.639185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.639413 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.639394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.641309 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.641284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.786415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.786329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:17:42.802408 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:42.802377    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"]
Sep 16 19:17:42.804074 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:17:42.804048    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740b3022_61d9_433c_9c51_7e9acb8345a2.slice/crio-5a0cc78339441f79dc90a5bde038c3e039b5987d79927e07e6c779ad94200132 WatchSource:0}: Error finding container 5a0cc78339441f79dc90a5bde038c3e039b5987d79927e07e6c779ad94200132: Status 404 returned error can't find the container with id 5a0cc78339441f79dc90a5bde038c3e039b5987d79927e07e6c779ad94200132
Sep 16 19:17:43.064922 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:43.064895    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"]
Sep 16 19:17:43.066889 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:17:43.066845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f6ba27_3f1a_4592_bd52_f4bb5582abbe.slice/crio-58a16552107110f18860c4f9dc1f7c4c3615f0cfe974cb65da05a794e1d68a21 WatchSource:0}: Error finding container 58a16552107110f18860c4f9dc1f7c4c3615f0cfe974cb65da05a794e1d68a21: Status 404 returned error can't find the container with id 58a16552107110f18860c4f9dc1f7c4c3615f0cfe974cb65da05a794e1d68a21
Sep 16 19:17:43.714159 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:43.714116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" event={"ID":"740b3022-61d9-433c-9c51-7e9acb8345a2","Type":"ContainerStarted","Data":"6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc"}
Sep 16 19:17:43.714635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:43.714220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" event={"ID":"740b3022-61d9-433c-9c51-7e9acb8345a2","Type":"ContainerStarted","Data":"5a0cc78339441f79dc90a5bde038c3e039b5987d79927e07e6c779ad94200132"}
Sep 16 19:17:43.715784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:43.715758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" event={"ID":"64f6ba27-3f1a-4592-bd52-f4bb5582abbe","Type":"ContainerStarted","Data":"e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9"}
Sep 16 19:17:43.715947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:43.715805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" event={"ID":"64f6ba27-3f1a-4592-bd52-f4bb5582abbe","Type":"ContainerStarted","Data":"58a16552107110f18860c4f9dc1f7c4c3615f0cfe974cb65da05a794e1d68a21"}
Sep 16 19:17:45.325415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:45.325376    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:45.344986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:45.344926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:45.730134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:45.730096    2581 generic.go:356] "Generic (PLEG): container finished" podID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerID="6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc" exitCode=0
Sep 16 19:17:45.730318 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:45.730165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" event={"ID":"740b3022-61d9-433c-9c51-7e9acb8345a2","Type":"ContainerDied","Data":"6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc"}
Sep 16 19:17:45.744084 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:45.744061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:17:46.740171 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:46.740134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" event={"ID":"740b3022-61d9-433c-9c51-7e9acb8345a2","Type":"ContainerStarted","Data":"e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"}
Sep 16 19:17:46.741772 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:46.741744    2581 generic.go:356] "Generic (PLEG): container finished" podID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerID="e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9" exitCode=0
Sep 16 19:17:46.741931 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:46.741815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" event={"ID":"64f6ba27-3f1a-4592-bd52-f4bb5582abbe","Type":"ContainerDied","Data":"e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9"}
Sep 16 19:17:46.785436 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:46.785391    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" podStartSLOduration=2.191418578 podStartE2EDuration="4.785376671s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.593958093s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:46.779561937 +0000 UTC m=+1968.083578927" watchObservedRunningTime="2026-09-16 19:17:46.785376671 +0000 UTC m=+1968.089393635"
Sep 16 19:17:47.529029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:47.528996    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:47.755259 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:47.755221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" event={"ID":"64f6ba27-3f1a-4592-bd52-f4bb5582abbe","Type":"ContainerStarted","Data":"ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb"}
Sep 16 19:17:47.774186 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:47.774130    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" podStartSLOduration=3.157584664 podStartE2EDuration="5.774114819s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.616530155s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:47.772949454 +0000 UTC m=+1969.076966426" watchObservedRunningTime="2026-09-16 19:17:47.774114819 +0000 UTC m=+1969.078131785"
Sep 16 19:17:52.528648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:52.528560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:52.541390 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:52.541360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:52.792344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:52.792256    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:17:57.756336 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:17:57.756302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:18:16.087419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:16.087389    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"]
Sep 16 19:18:17.768158 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:17.768129    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:18:29.008751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:29.008706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr_64f6ba27-3f1a-4592-bd52-f4bb5582abbe/main/0.log"
Sep 16 19:18:31.105765 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.105701    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6" podUID="ee95ac1f-ab40-4d0b-b3c4-79292b997c92" containerName="main" containerID="cri-o://fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7" gracePeriod=45
Sep 16 19:18:31.346048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.346019    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:18:31.420965 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.420939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-model-cache\") pod \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") "
Sep 16 19:18:31.421112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421012    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tmp-dir\") pod \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") "
Sep 16 19:18:31.421112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh2kw\" (UniqueName: \"kubernetes.io/projected/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-kube-api-access-kh2kw\") pod \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") "
Sep 16 19:18:31.421112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421050    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-home\") pod \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") "
Sep 16 19:18:31.421112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421080    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tls-certs\") pod \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") "
Sep 16 19:18:31.421337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-dshm\") pod \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\" (UID: \"ee95ac1f-ab40-4d0b-b3c4-79292b997c92\") "
Sep 16 19:18:31.421337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee95ac1f-ab40-4d0b-b3c4-79292b997c92" (UID: "ee95ac1f-ab40-4d0b-b3c4-79292b997c92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:31.421337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421283    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee95ac1f-ab40-4d0b-b3c4-79292b997c92" (UID: "ee95ac1f-ab40-4d0b-b3c4-79292b997c92"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:31.421337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421316    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-home" (OuterVolumeSpecName: "home") pod "ee95ac1f-ab40-4d0b-b3c4-79292b997c92" (UID: "ee95ac1f-ab40-4d0b-b3c4-79292b997c92"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:31.421549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421421    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:31.421549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421438    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:31.421549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.421449    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:31.423314 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.423284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-dshm" (OuterVolumeSpecName: "dshm") pod "ee95ac1f-ab40-4d0b-b3c4-79292b997c92" (UID: "ee95ac1f-ab40-4d0b-b3c4-79292b997c92"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:31.423403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.423284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee95ac1f-ab40-4d0b-b3c4-79292b997c92" (UID: "ee95ac1f-ab40-4d0b-b3c4-79292b997c92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:31.423403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.423311    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-kube-api-access-kh2kw" (OuterVolumeSpecName: "kube-api-access-kh2kw") pod "ee95ac1f-ab40-4d0b-b3c4-79292b997c92" (UID: "ee95ac1f-ab40-4d0b-b3c4-79292b997c92"). InnerVolumeSpecName "kube-api-access-kh2kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:31.522590 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.522566    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:31.522590 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.522588    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh2kw\" (UniqueName: \"kubernetes.io/projected/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-kube-api-access-kh2kw\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:31.522738 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.522599    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee95ac1f-ab40-4d0b-b3c4-79292b997c92-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:31.988353 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.988321    2581 generic.go:356] "Generic (PLEG): container finished" podID="ee95ac1f-ab40-4d0b-b3c4-79292b997c92" containerID="fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7" exitCode=0
Sep 16 19:18:31.988509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.988368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"
Sep 16 19:18:31.988509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.988413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6" event={"ID":"ee95ac1f-ab40-4d0b-b3c4-79292b997c92","Type":"ContainerDied","Data":"fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7"}
Sep 16 19:18:31.988509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.988462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6" event={"ID":"ee95ac1f-ab40-4d0b-b3c4-79292b997c92","Type":"ContainerDied","Data":"3c42f62bedda1673d3caacd407f307d05353db9a2af63dc293772d98c436979d"}
Sep 16 19:18:31.988509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.988485    2581 scope.go:117] "RemoveContainer" containerID="fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7"
Sep 16 19:18:31.998516 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.998501    2581 scope.go:117] "RemoveContainer" containerID="fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7"
Sep 16 19:18:31.998785 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:31.998766    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7\": container with ID starting with fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7 not found: ID does not exist" containerID="fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7"
Sep 16 19:18:31.998905 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:31.998788    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7"} err="failed to get container status \"fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7\": rpc error: code = NotFound desc = could not find container \"fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7\": container with ID starting with fc0613cb56d262a609718b249ff36f477eeb7d8114cf73b5990351291ca858d7 not found: ID does not exist"
Sep 16 19:18:32.011838 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:32.011814    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"]
Sep 16 19:18:32.015695 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:32.015670    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-7mbh6"]
Sep 16 19:18:33.222510 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:33.222477    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee95ac1f-ab40-4d0b-b3c4-79292b997c92" path="/var/lib/kubelet/pods/ee95ac1f-ab40-4d0b-b3c4-79292b997c92/volumes"
Sep 16 19:18:55.678952 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:55.678918    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"]
Sep 16 19:18:55.679539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:55.679130    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerName="main" containerID="cri-o://ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb" gracePeriod=30
Sep 16 19:18:55.686402 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:55.686373    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"]
Sep 16 19:18:55.695440 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:55.695407    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:18:55.695440 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:18:55.695440 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:18:55.695440 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa" cmd=["/bin/sleep","15"]
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:55.695463    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" message="" err=<
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:55.695480    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="main" containerID="cri-o://e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"
Sep 16 19:18:55.695652 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:55.695521    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="main" containerID="cri-o://e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa" gracePeriod=60
Sep 16 19:18:55.781800 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:55.781758    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Sep 16 19:18:56.046798 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.046774    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:18:56.115202 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.115163    2581 generic.go:356] "Generic (PLEG): container finished" podID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerID="e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa" exitCode=0
Sep 16 19:18:56.115366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.115217    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"
Sep 16 19:18:56.115366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.115251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" event={"ID":"740b3022-61d9-433c-9c51-7e9acb8345a2","Type":"ContainerDied","Data":"e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"}
Sep 16 19:18:56.115366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.115309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5" event={"ID":"740b3022-61d9-433c-9c51-7e9acb8345a2","Type":"ContainerDied","Data":"5a0cc78339441f79dc90a5bde038c3e039b5987d79927e07e6c779ad94200132"}
Sep 16 19:18:56.115366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.115334    2581 scope.go:117] "RemoveContainer" containerID="e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"
Sep 16 19:18:56.119045 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-tmp-dir\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119180 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119071    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-home\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119180 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119097    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-model-cache\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119180 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-dshm\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119179    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7rxt\" (UniqueName: \"kubernetes.io/projected/740b3022-61d9-433c-9c51-7e9acb8345a2-kube-api-access-k7rxt\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119229    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/740b3022-61d9-433c-9c51-7e9acb8345a2-cabundle-cert\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740b3022-61d9-433c-9c51-7e9acb8345a2-tls-certs\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-kserve-provision-location\") pod \"740b3022-61d9-433c-9c51-7e9acb8345a2\" (UID: \"740b3022-61d9-433c-9c51-7e9acb8345a2\") "
Sep 16 19:18:56.119645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-home" (OuterVolumeSpecName: "home") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.119645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.119645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119381    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.119819 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119751    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.119819 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119772    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.119819 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119787    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.120020 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.119946    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/740b3022-61d9-433c-9c51-7e9acb8345a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:56.121995 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.121969    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/740b3022-61d9-433c-9c51-7e9acb8345a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:56.122168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.122141    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/740b3022-61d9-433c-9c51-7e9acb8345a2-kube-api-access-k7rxt" (OuterVolumeSpecName: "kube-api-access-k7rxt") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "kube-api-access-k7rxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:56.122284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.122265    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-dshm" (OuterVolumeSpecName: "dshm") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.125865 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.125834    2581 scope.go:117] "RemoveContainer" containerID="6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc"
Sep 16 19:18:56.141658 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.141635    2581 scope.go:117] "RemoveContainer" containerID="e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"
Sep 16 19:18:56.141991 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:56.141970    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa\": container with ID starting with e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa not found: ID does not exist" containerID="e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"
Sep 16 19:18:56.142078 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.141999    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa"} err="failed to get container status \"e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa\": rpc error: code = NotFound desc = could not find container \"e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa\": container with ID starting with e993640289e0f216408e1cefbe92b7202a15e009bececdafd77284a9e3cccbfa not found: ID does not exist"
Sep 16 19:18:56.142078 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.142015    2581 scope.go:117] "RemoveContainer" containerID="6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc"
Sep 16 19:18:56.142303 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:56.142287    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc\": container with ID starting with 6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc not found: ID does not exist" containerID="6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc"
Sep 16 19:18:56.142361 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.142308    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc"} err="failed to get container status \"6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc\": rpc error: code = NotFound desc = could not find container \"6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc\": container with ID starting with 6b4314cb7ba31e7cf91e3ea451bc71dd671f0af1ad97306d8a60611f21bbb1bc not found: ID does not exist"
Sep 16 19:18:56.143671 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.143642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "740b3022-61d9-433c-9c51-7e9acb8345a2" (UID: "740b3022-61d9-433c-9c51-7e9acb8345a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.220410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.220323    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.220410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.220353    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7rxt\" (UniqueName: \"kubernetes.io/projected/740b3022-61d9-433c-9c51-7e9acb8345a2-kube-api-access-k7rxt\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.220410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.220366    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/740b3022-61d9-433c-9c51-7e9acb8345a2-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.220410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.220381    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740b3022-61d9-433c-9c51-7e9acb8345a2-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.220410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.220392    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/740b3022-61d9-433c-9c51-7e9acb8345a2-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.308311 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.308286    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:18:56.421663 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.421627    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-cabundle-cert\") pod \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") "
Sep 16 19:18:56.421848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.421681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tls-certs\") pod \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") "
Sep 16 19:18:56.421848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.421711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tmp-dir\") pod \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") "
Sep 16 19:18:56.421848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.421740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-kserve-provision-location\") pod \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\" (UID: \"64f6ba27-3f1a-4592-bd52-f4bb5582abbe\") "
Sep 16 19:18:56.422114 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.422088    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "64f6ba27-3f1a-4592-bd52-f4bb5582abbe" (UID: "64f6ba27-3f1a-4592-bd52-f4bb5582abbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:18:56.422176 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.422146    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64f6ba27-3f1a-4592-bd52-f4bb5582abbe" (UID: "64f6ba27-3f1a-4592-bd52-f4bb5582abbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.424176 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.424151    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64f6ba27-3f1a-4592-bd52-f4bb5582abbe" (UID: "64f6ba27-3f1a-4592-bd52-f4bb5582abbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:56.441763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.441737    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"]
Sep 16 19:18:56.445419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.445394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "64f6ba27-3f1a-4592-bd52-f4bb5582abbe" (UID: "64f6ba27-3f1a-4592-bd52-f4bb5582abbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:56.445821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.445802    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56579b9954-tkbl5"]
Sep 16 19:18:56.523296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.523218    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.523296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.523248    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.523296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.523257    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:56.523296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:56.523266    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64f6ba27-3f1a-4592-bd52-f4bb5582abbe-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:18:57.122631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.122550    2581 generic.go:356] "Generic (PLEG): container finished" podID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerID="ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb" exitCode=0
Sep 16 19:18:57.122631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.122600    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"
Sep 16 19:18:57.122631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.122616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" event={"ID":"64f6ba27-3f1a-4592-bd52-f4bb5582abbe","Type":"ContainerDied","Data":"ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb"}
Sep 16 19:18:57.123147 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.122672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr" event={"ID":"64f6ba27-3f1a-4592-bd52-f4bb5582abbe","Type":"ContainerDied","Data":"58a16552107110f18860c4f9dc1f7c4c3615f0cfe974cb65da05a794e1d68a21"}
Sep 16 19:18:57.123147 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.122700    2581 scope.go:117] "RemoveContainer" containerID="ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb"
Sep 16 19:18:57.131603 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.131585    2581 scope.go:117] "RemoveContainer" containerID="e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9"
Sep 16 19:18:57.138940 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.138918    2581 scope.go:117] "RemoveContainer" containerID="ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb"
Sep 16 19:18:57.139186 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:57.139170    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb\": container with ID starting with ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb not found: ID does not exist" containerID="ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb"
Sep 16 19:18:57.139229 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.139193    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb"} err="failed to get container status \"ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb\": rpc error: code = NotFound desc = could not find container \"ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb\": container with ID starting with ec41f4d702136148b4cccd97528f59e3ca43fd78a7e1e0f36bef4c8194d14adb not found: ID does not exist"
Sep 16 19:18:57.139229 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.139209    2581 scope.go:117] "RemoveContainer" containerID="e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9"
Sep 16 19:18:57.139453 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:18:57.139429    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9\": container with ID starting with e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9 not found: ID does not exist" containerID="e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9"
Sep 16 19:18:57.139499 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.139464    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9"} err="failed to get container status \"e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9\": rpc error: code = NotFound desc = could not find container \"e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9\": container with ID starting with e8a7c748def6947447098bb6abebfacd4fd8523941fb8404fd2e4fe4bebc35d9 not found: ID does not exist"
Sep 16 19:18:57.146661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.146640    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"]
Sep 16 19:18:57.150508 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.150490    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-7bdbd644d8-9hmcr"]
Sep 16 19:18:57.222880 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.222800    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" path="/var/lib/kubelet/pods/64f6ba27-3f1a-4592-bd52-f4bb5582abbe/volumes"
Sep 16 19:18:57.223547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:18:57.223518    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" path="/var/lib/kubelet/pods/740b3022-61d9-433c-9c51-7e9acb8345a2/volumes"
Sep 16 19:19:09.410605 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410566    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"]
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410893    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee95ac1f-ab40-4d0b-b3c4-79292b997c92" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410904    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee95ac1f-ab40-4d0b-b3c4-79292b997c92" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerName="storage-initializer"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerName="storage-initializer"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410932    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410937    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410945    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410950    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410956    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="storage-initializer"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.410961    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="storage-initializer"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.411005    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee95ac1f-ab40-4d0b-b3c4-79292b997c92" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.411012    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f6ba27-3f1a-4592-bd52-f4bb5582abbe" containerName="main"
Sep 16 19:19:09.411021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.411019    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="740b3022-61d9-433c-9c51-7e9acb8345a2" containerName="main"
Sep 16 19:19:09.413837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.413821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.417915 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.417890    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 16 19:19:09.417915 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.417908    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-62x4j\""
Sep 16 19:19:09.418098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.417929    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:19:09.418098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.417987    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:19:09.425808 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.425789    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"]
Sep 16 19:19:09.529135 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.529105    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtrs\" (UniqueName: \"kubernetes.io/projected/9b1c888b-225c-4ddb-83b8-07918eecd6c1-kube-api-access-kmtrs\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.529316 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.529167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-dshm\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.529316 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.529232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.529316 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.529305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tls-certs\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.529451 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.529340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-home\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.529451 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.529392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-model-cache\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.630790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.630753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tls-certs\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.630790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.630787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-home\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.630821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-model-cache\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.630844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtrs\" (UniqueName: \"kubernetes.io/projected/9b1c888b-225c-4ddb-83b8-07918eecd6c1-kube-api-access-kmtrs\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.630915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-dshm\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.630953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631246 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.631228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-model-cache\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631305 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.631267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-home\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.631305 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.631294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.633108 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.633086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-dshm\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.633356 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.633339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tls-certs\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.639568 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.639548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtrs\" (UniqueName: \"kubernetes.io/projected/9b1c888b-225c-4ddb-83b8-07918eecd6c1-kube-api-access-kmtrs\") pod \"stop-v1-kserve-5657b88c95-c7lnx\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:09.723385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:09.723308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:10.003950 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:10.003917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"]
Sep 16 19:19:10.005718 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:19:10.005684    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1c888b_225c_4ddb_83b8_07918eecd6c1.slice/crio-8f65fd2ad650a0685194174f528c6b89f55794654b5befac12fff6717323f5db WatchSource:0}: Error finding container 8f65fd2ad650a0685194174f528c6b89f55794654b5befac12fff6717323f5db: Status 404 returned error can't find the container with id 8f65fd2ad650a0685194174f528c6b89f55794654b5befac12fff6717323f5db
Sep 16 19:19:10.187104 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:10.187069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx" event={"ID":"9b1c888b-225c-4ddb-83b8-07918eecd6c1","Type":"ContainerStarted","Data":"2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211"}
Sep 16 19:19:10.187272 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:10.187123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx" event={"ID":"9b1c888b-225c-4ddb-83b8-07918eecd6c1","Type":"ContainerStarted","Data":"8f65fd2ad650a0685194174f528c6b89f55794654b5befac12fff6717323f5db"}
Sep 16 19:19:10.207886 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:10.207828    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx" podStartSLOduration=1.207813346 podStartE2EDuration="1.207813346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:10.205648618 +0000 UTC m=+2051.509665584" watchObservedRunningTime="2026-09-16 19:19:10.207813346 +0000 UTC m=+2051.511830310"
Sep 16 19:19:10.724183 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:10.724143    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:19.723490 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:19.723456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:19.737155 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:19.737131    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:20.247504 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:20.247471    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:19:48.717911 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.717877    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"]
Sep 16 19:19:48.721679 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.721657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.724522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.724497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 19:19:48.724621 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.724497    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:19:48.725619 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.725604    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:19:48.725686 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.725642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-cv956\""
Sep 16 19:19:48.725743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.725718    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:19:48.725806 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.725746    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:19:48.733249 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.733230    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"]
Sep 16 19:19:48.835393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-dshm\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d67580-4e97-4b03-acb5-3caf34d16a70-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-home\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df82p\" (UniqueName: \"kubernetes.io/projected/c3d67580-4e97-4b03-acb5-3caf34d16a70-kube-api-access-df82p\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d67580-4e97-4b03-acb5-3caf34d16a70-tls-certs\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.835843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.835666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-model-cache\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d67580-4e97-4b03-acb5-3caf34d16a70-tls-certs\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-model-cache\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-dshm\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d67580-4e97-4b03-acb5-3caf34d16a70-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.936751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-home\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.937032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df82p\" (UniqueName: \"kubernetes.io/projected/c3d67580-4e97-4b03-acb5-3caf34d16a70-kube-api-access-df82p\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.937032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.936995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.937032 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.937025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-home\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.937204 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.937128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.937262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.937208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-model-cache\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.937375 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.937355    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d67580-4e97-4b03-acb5-3caf34d16a70-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.938877 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.938830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-dshm\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.939103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.939087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d67580-4e97-4b03-acb5-3caf34d16a70-tls-certs\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.948138 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.948114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df82p\" (UniqueName: \"kubernetes.io/projected/c3d67580-4e97-4b03-acb5-3caf34d16a70-kube-api-access-df82p\") pod \"tokenizer-template-args-test-kserve-7d598c7594-qbxbf\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:48.965495 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.965469    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"]
Sep 16 19:19:48.970203 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.970148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:48.982028 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:48.982001    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"]
Sep 16 19:19:49.032729 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.032697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:49.037822 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.037799    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/066dd813-2aff-4349-9e18-0ecdfeb4123d-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.037944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.037833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066dd813-2aff-4349-9e18-0ecdfeb4123d-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.037944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.037886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.037944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.037919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.138817 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.138789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.138982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.138825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.138982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.138922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/066dd813-2aff-4349-9e18-0ecdfeb4123d-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.138982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.138958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066dd813-2aff-4349-9e18-0ecdfeb4123d-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.139286 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.139263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.139325 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.139276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.139569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.139545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066dd813-2aff-4349-9e18-0ecdfeb4123d-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.141478 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.141453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/066dd813-2aff-4349-9e18-0ecdfeb4123d-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.283635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.283551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:49.307522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.307496    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"]
Sep 16 19:19:49.309750 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:19:49.309724    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d67580_4e97_4b03_acb5_3caf34d16a70.slice/crio-f83ffff80a378eda375c83d757211d5d74a9b08c90bea9954f2c5c9cea9ac1d1 WatchSource:0}: Error finding container f83ffff80a378eda375c83d757211d5d74a9b08c90bea9954f2c5c9cea9ac1d1: Status 404 returned error can't find the container with id f83ffff80a378eda375c83d757211d5d74a9b08c90bea9954f2c5c9cea9ac1d1
Sep 16 19:19:49.379557 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.379527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" event={"ID":"c3d67580-4e97-4b03-acb5-3caf34d16a70","Type":"ContainerStarted","Data":"dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539"}
Sep 16 19:19:49.379773 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.379615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" event={"ID":"c3d67580-4e97-4b03-acb5-3caf34d16a70","Type":"ContainerStarted","Data":"f83ffff80a378eda375c83d757211d5d74a9b08c90bea9954f2c5c9cea9ac1d1"}
Sep 16 19:19:49.562683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:49.562657    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"]
Sep 16 19:19:49.565055 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:19:49.565021    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066dd813_2aff_4349_9e18_0ecdfeb4123d.slice/crio-62e0fd9bc9fceab79cdd4e24883fb57e9bae834bd8550415f440bb215bfc9f96 WatchSource:0}: Error finding container 62e0fd9bc9fceab79cdd4e24883fb57e9bae834bd8550415f440bb215bfc9f96: Status 404 returned error can't find the container with id 62e0fd9bc9fceab79cdd4e24883fb57e9bae834bd8550415f440bb215bfc9f96
Sep 16 19:19:50.386977 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:50.386937    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" event={"ID":"066dd813-2aff-4349-9e18-0ecdfeb4123d","Type":"ContainerStarted","Data":"f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9"}
Sep 16 19:19:50.387370 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:50.386995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" event={"ID":"066dd813-2aff-4349-9e18-0ecdfeb4123d","Type":"ContainerStarted","Data":"62e0fd9bc9fceab79cdd4e24883fb57e9bae834bd8550415f440bb215bfc9f96"}
Sep 16 19:19:52.400751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:52.400715    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerID="dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539" exitCode=0
Sep 16 19:19:52.401221 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:52.400754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" event={"ID":"c3d67580-4e97-4b03-acb5-3caf34d16a70","Type":"ContainerDied","Data":"dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539"}
Sep 16 19:19:52.402466 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:52.402444    2581 generic.go:356] "Generic (PLEG): container finished" podID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerID="f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9" exitCode=0
Sep 16 19:19:52.402579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:52.402491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" event={"ID":"066dd813-2aff-4349-9e18-0ecdfeb4123d","Type":"ContainerDied","Data":"f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9"}
Sep 16 19:19:53.410821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:53.410782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" event={"ID":"066dd813-2aff-4349-9e18-0ecdfeb4123d","Type":"ContainerStarted","Data":"759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269"}
Sep 16 19:19:53.411285 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:53.411030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:19:53.412800 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:53.412779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" event={"ID":"c3d67580-4e97-4b03-acb5-3caf34d16a70","Type":"ContainerStarted","Data":"134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"}
Sep 16 19:19:53.430162 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:53.430121    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" podStartSLOduration=2.851912239 podStartE2EDuration="5.430107629s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.57819539s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:53.428199621 +0000 UTC m=+2094.732216601" watchObservedRunningTime="2026-09-16 19:19:53.430107629 +0000 UTC m=+2094.734124596"
Sep 16 19:19:53.448479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:53.448442    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" podStartSLOduration=2.875622625 podStartE2EDuration="5.448430797s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.572808172s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:53.447242997 +0000 UTC m=+2094.751259964" watchObservedRunningTime="2026-09-16 19:19:53.448430797 +0000 UTC m=+2094.752447764"
Sep 16 19:19:54.033185 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:54.033147    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:59.033028 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.032987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:59.046566 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.046543    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:19:59.336316 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.336227    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:19:59.338898 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.338876    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:19:59.339349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.339328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:19:59.342124 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.342102    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:19:59.458616 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:19:59.458589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:20:24.430275 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:24.430236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:20:29.927920 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:29.927881    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"]
Sep 16 19:20:32.331980 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:32.331947    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7_066dd813-2aff-4349-9e18-0ecdfeb4123d/main/0.log"
Sep 16 19:20:44.950568 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:44.950525    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx" podUID="9b1c888b-225c-4ddb-83b8-07918eecd6c1" containerName="main" containerID="cri-o://2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211" gracePeriod=45
Sep 16 19:20:45.205881 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.205797    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:20:45.324820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.324771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tls-certs\") pod \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") "
Sep 16 19:20:45.324820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.324831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tmp-dir\") pod \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") "
Sep 16 19:20:45.325112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.324852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-home\") pod \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") "
Sep 16 19:20:45.325112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.324936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmtrs\" (UniqueName: \"kubernetes.io/projected/9b1c888b-225c-4ddb-83b8-07918eecd6c1-kube-api-access-kmtrs\") pod \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") "
Sep 16 19:20:45.325112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.324982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-model-cache\") pod \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") "
Sep 16 19:20:45.325112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-dshm\") pod \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\" (UID: \"9b1c888b-225c-4ddb-83b8-07918eecd6c1\") "
Sep 16 19:20:45.325329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b1c888b-225c-4ddb-83b8-07918eecd6c1" (UID: "9b1c888b-225c-4ddb-83b8-07918eecd6c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:45.325329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-home" (OuterVolumeSpecName: "home") pod "9b1c888b-225c-4ddb-83b8-07918eecd6c1" (UID: "9b1c888b-225c-4ddb-83b8-07918eecd6c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:45.325329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325275    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b1c888b-225c-4ddb-83b8-07918eecd6c1" (UID: "9b1c888b-225c-4ddb-83b8-07918eecd6c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:45.325483 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325411    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.325483 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325426    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.325483 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.325435    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.327215 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.327179    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1c888b-225c-4ddb-83b8-07918eecd6c1-kube-api-access-kmtrs" (OuterVolumeSpecName: "kube-api-access-kmtrs") pod "9b1c888b-225c-4ddb-83b8-07918eecd6c1" (UID: "9b1c888b-225c-4ddb-83b8-07918eecd6c1"). InnerVolumeSpecName "kube-api-access-kmtrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:20:45.327340 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.327249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b1c888b-225c-4ddb-83b8-07918eecd6c1" (UID: "9b1c888b-225c-4ddb-83b8-07918eecd6c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:20:45.327340 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.327320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-dshm" (OuterVolumeSpecName: "dshm") pod "9b1c888b-225c-4ddb-83b8-07918eecd6c1" (UID: "9b1c888b-225c-4ddb-83b8-07918eecd6c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:20:45.426395 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.426360    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtrs\" (UniqueName: \"kubernetes.io/projected/9b1c888b-225c-4ddb-83b8-07918eecd6c1-kube-api-access-kmtrs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.426395 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.426388    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b1c888b-225c-4ddb-83b8-07918eecd6c1-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.426395 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.426401    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b1c888b-225c-4ddb-83b8-07918eecd6c1-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:20:45.679947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.679912    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b1c888b-225c-4ddb-83b8-07918eecd6c1" containerID="2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211" exitCode=0
Sep 16 19:20:45.680139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.679960    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"
Sep 16 19:20:45.680139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.679998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx" event={"ID":"9b1c888b-225c-4ddb-83b8-07918eecd6c1","Type":"ContainerDied","Data":"2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211"}
Sep 16 19:20:45.680139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.680061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx" event={"ID":"9b1c888b-225c-4ddb-83b8-07918eecd6c1","Type":"ContainerDied","Data":"8f65fd2ad650a0685194174f528c6b89f55794654b5befac12fff6717323f5db"}
Sep 16 19:20:45.680139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.680093    2581 scope.go:117] "RemoveContainer" containerID="2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211"
Sep 16 19:20:45.691838 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.691819    2581 scope.go:117] "RemoveContainer" containerID="2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211"
Sep 16 19:20:45.692167 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:20:45.692146    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211\": container with ID starting with 2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211 not found: ID does not exist" containerID="2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211"
Sep 16 19:20:45.692231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.692176    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211"} err="failed to get container status \"2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211\": rpc error: code = NotFound desc = could not find container \"2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211\": container with ID starting with 2678017bfd2ee820acec2a858f6daa5d704640de04a3dc8f3a77a102ed7be211 not found: ID does not exist"
Sep 16 19:20:45.704871 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.704835    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"]
Sep 16 19:20:45.709563 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:45.709540    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-c7lnx"]
Sep 16 19:20:47.221690 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:20:47.221657    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b1c888b-225c-4ddb-83b8-07918eecd6c1" path="/var/lib/kubelet/pods/9b1c888b-225c-4ddb-83b8-07918eecd6c1/volumes"
Sep 16 19:21:26.071959 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.071917    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"]
Sep 16 19:21:26.077594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.077567    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"]
Sep 16 19:21:26.077912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.077789    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerName="main" containerID="cri-o://759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269" gracePeriod=30
Sep 16 19:21:26.081499 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:21:26.081465    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:21:26.081499 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:26.081499 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:26.081499 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd" cmd=["/bin/sleep","15"]
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.081521    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" message="" err=<
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:21:26.081544    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="main" containerID="cri-o://134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"
Sep 16 19:21:26.082000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.081585    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="main" containerID="cri-o://134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd" gracePeriod=60
Sep 16 19:21:26.444146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.444120    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:21:26.461264 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-model-cache\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-kserve-provision-location\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461325    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-tmp-dir\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-df82p\" (UniqueName: \"kubernetes.io/projected/c3d67580-4e97-4b03-acb5-3caf34d16a70-kube-api-access-df82p\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-home\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-dshm\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461664 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d67580-4e97-4b03-acb5-3caf34d16a70-tls-certs\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461664 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-model-cache" (OuterVolumeSpecName: "model-cache") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.461664 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461529    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d67580-4e97-4b03-acb5-3caf34d16a70-cabundle-cert\") pod \"c3d67580-4e97-4b03-acb5-3caf34d16a70\" (UID: \"c3d67580-4e97-4b03-acb5-3caf34d16a70\") "
Sep 16 19:21:26.461664 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461553    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.461664 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-home" (OuterVolumeSpecName: "home") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.461946 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461791    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.461946 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461813    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.461946 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461827    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.462103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.461961    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d67580-4e97-4b03-acb5-3caf34d16a70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:26.463982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.463712    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-dshm" (OuterVolumeSpecName: "dshm") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.463982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.463936    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d67580-4e97-4b03-acb5-3caf34d16a70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:26.464155 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.463996    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d67580-4e97-4b03-acb5-3caf34d16a70-kube-api-access-df82p" (OuterVolumeSpecName: "kube-api-access-df82p") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "kube-api-access-df82p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:26.484156 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.484128    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c3d67580-4e97-4b03-acb5-3caf34d16a70" (UID: "c3d67580-4e97-4b03-acb5-3caf34d16a70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.562472 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.562435    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3d67580-4e97-4b03-acb5-3caf34d16a70-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.562472 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.562470    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c3d67580-4e97-4b03-acb5-3caf34d16a70-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.562723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.562484    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.562723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.562497    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df82p\" (UniqueName: \"kubernetes.io/projected/c3d67580-4e97-4b03-acb5-3caf34d16a70-kube-api-access-df82p\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.562723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.562515    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c3d67580-4e97-4b03-acb5-3caf34d16a70-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.627447 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.627423    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:21:26.662924 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.662894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066dd813-2aff-4349-9e18-0ecdfeb4123d-cabundle-cert\") pod \"066dd813-2aff-4349-9e18-0ecdfeb4123d\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") "
Sep 16 19:21:26.663067 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.662942    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/066dd813-2aff-4349-9e18-0ecdfeb4123d-tls-certs\") pod \"066dd813-2aff-4349-9e18-0ecdfeb4123d\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") "
Sep 16 19:21:26.663067 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.663051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-tmp-dir\") pod \"066dd813-2aff-4349-9e18-0ecdfeb4123d\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") "
Sep 16 19:21:26.663146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.663087    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-kserve-provision-location\") pod \"066dd813-2aff-4349-9e18-0ecdfeb4123d\" (UID: \"066dd813-2aff-4349-9e18-0ecdfeb4123d\") "
Sep 16 19:21:26.663393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.663362    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066dd813-2aff-4349-9e18-0ecdfeb4123d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "066dd813-2aff-4349-9e18-0ecdfeb4123d" (UID: "066dd813-2aff-4349-9e18-0ecdfeb4123d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:26.663503 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.663466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "066dd813-2aff-4349-9e18-0ecdfeb4123d" (UID: "066dd813-2aff-4349-9e18-0ecdfeb4123d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.665479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.665462    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066dd813-2aff-4349-9e18-0ecdfeb4123d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "066dd813-2aff-4349-9e18-0ecdfeb4123d" (UID: "066dd813-2aff-4349-9e18-0ecdfeb4123d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:26.686418 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.686382    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "066dd813-2aff-4349-9e18-0ecdfeb4123d" (UID: "066dd813-2aff-4349-9e18-0ecdfeb4123d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:26.764064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.764028    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/066dd813-2aff-4349-9e18-0ecdfeb4123d-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.764064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.764060    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.764219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.764070    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066dd813-2aff-4349-9e18-0ecdfeb4123d-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.764219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.764079    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066dd813-2aff-4349-9e18-0ecdfeb4123d-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:21:26.887639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.887552    2581 generic.go:356] "Generic (PLEG): container finished" podID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerID="759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269" exitCode=0
Sep 16 19:21:26.887639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.887603    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"
Sep 16 19:21:26.887897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.887635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" event={"ID":"066dd813-2aff-4349-9e18-0ecdfeb4123d","Type":"ContainerDied","Data":"759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269"}
Sep 16 19:21:26.887897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.887684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7" event={"ID":"066dd813-2aff-4349-9e18-0ecdfeb4123d","Type":"ContainerDied","Data":"62e0fd9bc9fceab79cdd4e24883fb57e9bae834bd8550415f440bb215bfc9f96"}
Sep 16 19:21:26.887897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.887709    2581 scope.go:117] "RemoveContainer" containerID="759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269"
Sep 16 19:21:26.889662 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.889633    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerID="134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd" exitCode=0
Sep 16 19:21:26.889804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.889677    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"
Sep 16 19:21:26.889804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.889739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" event={"ID":"c3d67580-4e97-4b03-acb5-3caf34d16a70","Type":"ContainerDied","Data":"134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"}
Sep 16 19:21:26.889969 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.889828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf" event={"ID":"c3d67580-4e97-4b03-acb5-3caf34d16a70","Type":"ContainerDied","Data":"f83ffff80a378eda375c83d757211d5d74a9b08c90bea9954f2c5c9cea9ac1d1"}
Sep 16 19:21:26.898255 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.898237    2581 scope.go:117] "RemoveContainer" containerID="f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9"
Sep 16 19:21:26.907624 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.907608    2581 scope.go:117] "RemoveContainer" containerID="759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269"
Sep 16 19:21:26.907903 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:21:26.907885    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269\": container with ID starting with 759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269 not found: ID does not exist" containerID="759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269"
Sep 16 19:21:26.907984 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.907915    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269"} err="failed to get container status \"759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269\": rpc error: code = NotFound desc = could not find container \"759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269\": container with ID starting with 759e642943a952d068cf008d4b4f999f2126d7a3d3275eaeeaef5b087d193269 not found: ID does not exist"
Sep 16 19:21:26.907984 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.907939    2581 scope.go:117] "RemoveContainer" containerID="f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9"
Sep 16 19:21:26.908226 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:21:26.908209    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9\": container with ID starting with f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9 not found: ID does not exist" containerID="f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9"
Sep 16 19:21:26.908293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.908234    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9"} err="failed to get container status \"f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9\": rpc error: code = NotFound desc = could not find container \"f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9\": container with ID starting with f663796c21e4b6be139b9d7f74fde802442e75cc4c145e2722f84bb3535c46a9 not found: ID does not exist"
Sep 16 19:21:26.908293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.908254    2581 scope.go:117] "RemoveContainer" containerID="134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"
Sep 16 19:21:26.916224 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.916198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"]
Sep 16 19:21:26.917389 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.917368    2581 scope.go:117] "RemoveContainer" containerID="dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539"
Sep 16 19:21:26.923649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.923629    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-7f6465bd8-8p2n7"]
Sep 16 19:21:26.925494 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.925476    2581 scope.go:117] "RemoveContainer" containerID="134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"
Sep 16 19:21:26.925756 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:21:26.925738    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd\": container with ID starting with 134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd not found: ID does not exist" containerID="134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"
Sep 16 19:21:26.925819 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.925774    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd"} err="failed to get container status \"134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd\": rpc error: code = NotFound desc = could not find container \"134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd\": container with ID starting with 134f57553a3ac8d3e3c7626be037e1acd85b4ee363ab202578bac697fbe8d5bd not found: ID does not exist"
Sep 16 19:21:26.925819 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.925792    2581 scope.go:117] "RemoveContainer" containerID="dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539"
Sep 16 19:21:26.926101 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:21:26.926077    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539\": container with ID starting with dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539 not found: ID does not exist" containerID="dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539"
Sep 16 19:21:26.926150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.926112    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539"} err="failed to get container status \"dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539\": rpc error: code = NotFound desc = could not find container \"dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539\": container with ID starting with dd0017b2b3bff6080af00928016a4b06e1cffc6faa4c9fb15e64a911e9b02539 not found: ID does not exist"
Sep 16 19:21:26.939946 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.939915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"]
Sep 16 19:21:26.945062 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:26.945041    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7d598c7594-qbxbf"]
Sep 16 19:21:27.222021 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:27.221991    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" path="/var/lib/kubelet/pods/066dd813-2aff-4349-9e18-0ecdfeb4123d/volumes"
Sep 16 19:21:27.222634 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:21:27.222613    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" path="/var/lib/kubelet/pods/c3d67580-4e97-4b03-acb5-3caf34d16a70/volumes"
Sep 16 19:22:46.002788 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.002748    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"]
Sep 16 19:22:46.003254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003237    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="storage-initializer"
Sep 16 19:22:46.003299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003256    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="storage-initializer"
Sep 16 19:22:46.003299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003277    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b1c888b-225c-4ddb-83b8-07918eecd6c1" containerName="main"
Sep 16 19:22:46.003299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003282    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b1c888b-225c-4ddb-83b8-07918eecd6c1" containerName="main"
Sep 16 19:22:46.003299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003289    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="main"
Sep 16 19:22:46.003299 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003294    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="main"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003303    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerName="storage-initializer"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003308    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerName="storage-initializer"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003313    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerName="main"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003319    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerName="main"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b1c888b-225c-4ddb-83b8-07918eecd6c1" containerName="main"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003410    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="066dd813-2aff-4349-9e18-0ecdfeb4123d" containerName="main"
Sep 16 19:22:46.003469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.003417    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3d67580-4e97-4b03-acb5-3caf34d16a70" containerName="main"
Sep 16 19:22:46.006333 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.006313    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.009694 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.009667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:22:46.010756 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.010734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-cjrz4\""
Sep 16 19:22:46.010876 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.010815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:22:46.010945 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.010827    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:22:46.017659 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.017638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"]
Sep 16 19:22:46.078013 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.077976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.078013 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.078019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.078255 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.078037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.078255 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.078129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt2mz\" (UniqueName: \"kubernetes.io/projected/88f86b8e-8a93-44d6-9e4d-022832f93f2a-kube-api-access-bt2mz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.078255 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.078167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.078255 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.078229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179333 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt2mz\" (UniqueName: \"kubernetes.io/projected/88f86b8e-8a93-44d6-9e4d-022832f93f2a-kube-api-access-bt2mz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179333 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179608 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179769 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.179930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.179896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.180123 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.180034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.180186 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.180149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.181705 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.181685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.182134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.182114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.189662 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.189640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt2mz\" (UniqueName: \"kubernetes.io/projected/88f86b8e-8a93-44d6-9e4d-022832f93f2a-kube-api-access-bt2mz\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.318019 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.317922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:46.592039 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:46.592011    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"]
Sep 16 19:22:46.594602 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:22:46.594572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f86b8e_8a93_44d6_9e4d_022832f93f2a.slice/crio-f2153c5001ed6eae41d05600f16000ca17e97752468707427dfb90a1306a2402 WatchSource:0}: Error finding container f2153c5001ed6eae41d05600f16000ca17e97752468707427dfb90a1306a2402: Status 404 returned error can't find the container with id f2153c5001ed6eae41d05600f16000ca17e97752468707427dfb90a1306a2402
Sep 16 19:22:47.279712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:47.279672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" event={"ID":"88f86b8e-8a93-44d6-9e4d-022832f93f2a","Type":"ContainerStarted","Data":"fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"}
Sep 16 19:22:47.280146 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:47.279729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" event={"ID":"88f86b8e-8a93-44d6-9e4d-022832f93f2a","Type":"ContainerStarted","Data":"f2153c5001ed6eae41d05600f16000ca17e97752468707427dfb90a1306a2402"}
Sep 16 19:22:47.305546 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:47.305490    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" podStartSLOduration=2.305472935 podStartE2EDuration="2.305472935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:47.303372387 +0000 UTC m=+2268.607389367" watchObservedRunningTime="2026-09-16 19:22:47.305472935 +0000 UTC m=+2268.609489902"
Sep 16 19:22:47.318456 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:47.318414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:56.318751 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:56.318712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:56.331954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:56.331926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:22:57.330912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:22:57.330880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:23:37.411610 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:37.411578    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"]
Sep 16 19:23:37.420829 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:23:37.420788    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:23:37.420829 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:37.420829 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:37.420829 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c" cmd=["/bin/sleep","15"]
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:37.420848    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" message="" err=<
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:23:37.420890    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" podUID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" containerName="main" containerID="cri-o://fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"
Sep 16 19:23:37.421131 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:37.420945    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" podUID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" containerName="main" containerID="cri-o://fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c" gracePeriod=60
Sep 16 19:23:38.155914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.155884    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:23:38.202506 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-home\") pod \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") "
Sep 16 19:23:38.202506 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202510    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-dshm\") pod \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") "
Sep 16 19:23:38.202725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202532    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt2mz\" (UniqueName: \"kubernetes.io/projected/88f86b8e-8a93-44d6-9e4d-022832f93f2a-kube-api-access-bt2mz\") pod \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") "
Sep 16 19:23:38.202725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-model-cache\") pod \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") "
Sep 16 19:23:38.202725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202631    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tmp-dir\") pod \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") "
Sep 16 19:23:38.202725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tls-certs\") pod \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\" (UID: \"88f86b8e-8a93-44d6-9e4d-022832f93f2a\") "
Sep 16 19:23:38.202976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202831    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-home" (OuterVolumeSpecName: "home") pod "88f86b8e-8a93-44d6-9e4d-022832f93f2a" (UID: "88f86b8e-8a93-44d6-9e4d-022832f93f2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:38.202976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.202891    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-model-cache" (OuterVolumeSpecName: "model-cache") pod "88f86b8e-8a93-44d6-9e4d-022832f93f2a" (UID: "88f86b8e-8a93-44d6-9e4d-022832f93f2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:38.203244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.203222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88f86b8e-8a93-44d6-9e4d-022832f93f2a" (UID: "88f86b8e-8a93-44d6-9e4d-022832f93f2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:38.204947 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.204918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f86b8e-8a93-44d6-9e4d-022832f93f2a-kube-api-access-bt2mz" (OuterVolumeSpecName: "kube-api-access-bt2mz") pod "88f86b8e-8a93-44d6-9e4d-022832f93f2a" (UID: "88f86b8e-8a93-44d6-9e4d-022832f93f2a"). InnerVolumeSpecName "kube-api-access-bt2mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:38.205115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.205093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-dshm" (OuterVolumeSpecName: "dshm") pod "88f86b8e-8a93-44d6-9e4d-022832f93f2a" (UID: "88f86b8e-8a93-44d6-9e4d-022832f93f2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:38.205181 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.205096    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88f86b8e-8a93-44d6-9e4d-022832f93f2a" (UID: "88f86b8e-8a93-44d6-9e4d-022832f93f2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:38.304207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.304137    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:38.304207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.304164    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f86b8e-8a93-44d6-9e4d-022832f93f2a-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:38.304207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.304173    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:38.304207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.304181    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:38.304207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.304190    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt2mz\" (UniqueName: \"kubernetes.io/projected/88f86b8e-8a93-44d6-9e4d-022832f93f2a-kube-api-access-bt2mz\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:38.304207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.304198    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f86b8e-8a93-44d6-9e4d-022832f93f2a-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:23:38.528250 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.528218    2581 generic.go:356] "Generic (PLEG): container finished" podID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" containerID="fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c" exitCode=0
Sep 16 19:23:38.528640 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.528262    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"
Sep 16 19:23:38.528640 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.528282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" event={"ID":"88f86b8e-8a93-44d6-9e4d-022832f93f2a","Type":"ContainerDied","Data":"fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"}
Sep 16 19:23:38.528640 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.528348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn" event={"ID":"88f86b8e-8a93-44d6-9e4d-022832f93f2a","Type":"ContainerDied","Data":"f2153c5001ed6eae41d05600f16000ca17e97752468707427dfb90a1306a2402"}
Sep 16 19:23:38.528640 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.528369    2581 scope.go:117] "RemoveContainer" containerID="fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"
Sep 16 19:23:38.537686 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.537665    2581 scope.go:117] "RemoveContainer" containerID="fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"
Sep 16 19:23:38.537958 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:23:38.537939    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c\": container with ID starting with fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c not found: ID does not exist" containerID="fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"
Sep 16 19:23:38.538017 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.537965    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c"} err="failed to get container status \"fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c\": rpc error: code = NotFound desc = could not find container \"fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c\": container with ID starting with fb9b6bd52999c360709de43873d25dc2963701c86cb239d7848865b450ae448c not found: ID does not exist"
Sep 16 19:23:38.550765 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.550744    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"]
Sep 16 19:23:38.556438 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:38.556388    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-s6tsn"]
Sep 16 19:23:39.221712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:39.221680    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" path="/var/lib/kubelet/pods/88f86b8e-8a93-44d6-9e4d-022832f93f2a/volumes"
Sep 16 19:23:52.410597 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.410507    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"]
Sep 16 19:23:52.411055 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.410903    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" containerName="main"
Sep 16 19:23:52.411055 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.410921    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" containerName="main"
Sep 16 19:23:52.411055 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.410993    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f86b8e-8a93-44d6-9e4d-022832f93f2a" containerName="main"
Sep 16 19:23:52.415283 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.415263    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.418580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.418559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-sgp6q\""
Sep 16 19:23:52.419906 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.419888    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 19:23:52.420006 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.419933    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 19:23:52.420006 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.419936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 19:23:52.427637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.427615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"]
Sep 16 19:23:52.520794 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.520762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-home\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.520794 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.520797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e0600f-67bf-4d09-80fa-7a83095b63e4-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.521042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.520937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.521042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.520994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.521042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.521018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4zb\" (UniqueName: \"kubernetes.io/projected/30e0600f-67bf-4d09-80fa-7a83095b63e4-kube-api-access-rk4zb\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.521161 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.521061    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622010 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.621974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622041    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4zb\" (UniqueName: \"kubernetes.io/projected/30e0600f-67bf-4d09-80fa-7a83095b63e4-kube-api-access-rk4zb\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-home\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e0600f-67bf-4d09-80fa-7a83095b63e4-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622467 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.622592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.622576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-home\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.624352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.624332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-dshm\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.624484 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.624468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e0600f-67bf-4d09-80fa-7a83095b63e4-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.630339 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.630310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4zb\" (UniqueName: \"kubernetes.io/projected/30e0600f-67bf-4d09-80fa-7a83095b63e4-kube-api-access-rk4zb\") pod \"tri-v1-kserve-79f8f98bc5-wkwvf\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.726449 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.726371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:23:52.997756 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:52.997733    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"]
Sep 16 19:23:52.999592 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:23:52.999565    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e0600f_67bf_4d09_80fa_7a83095b63e4.slice/crio-71f1575903ef7e416bfe6c841c9475b98ff755b38a1a77f8126db623f42a8dbe WatchSource:0}: Error finding container 71f1575903ef7e416bfe6c841c9475b98ff755b38a1a77f8126db623f42a8dbe: Status 404 returned error can't find the container with id 71f1575903ef7e416bfe6c841c9475b98ff755b38a1a77f8126db623f42a8dbe
Sep 16 19:23:53.603224 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:53.603188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf" event={"ID":"30e0600f-67bf-4d09-80fa-7a83095b63e4","Type":"ContainerStarted","Data":"580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279"}
Sep 16 19:23:53.603614 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:53.603253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf" event={"ID":"30e0600f-67bf-4d09-80fa-7a83095b63e4","Type":"ContainerStarted","Data":"71f1575903ef7e416bfe6c841c9475b98ff755b38a1a77f8126db623f42a8dbe"}
Sep 16 19:23:53.623145 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:53.623098    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf" podStartSLOduration=1.6230812000000001 podStartE2EDuration="1.6230812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:53.622336266 +0000 UTC m=+2334.926353223" watchObservedRunningTime="2026-09-16 19:23:53.6230812 +0000 UTC m=+2334.927098167"
Sep 16 19:23:53.726519 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:23:53.726484    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:24:02.726992 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:02.726960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:24:02.739471 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:02.739446    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:24:03.663671 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:03.663635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:24:49.490635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.490600    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"]
Sep 16 19:24:49.493848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.493826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.496685 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.496663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:24:49.497870 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.497841    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:24:49.498037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.497954    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:24:49.498037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.497971    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-x4kjb\""
Sep 16 19:24:49.504174 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.504152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"]
Sep 16 19:24:49.571505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.571478    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.571628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.571514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.571628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.571544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.571628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.571606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.571752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.571641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7c2\" (UniqueName: \"kubernetes.io/projected/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-kube-api-access-pf7c2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.571752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.571667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672806 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7c2\" (UniqueName: \"kubernetes.io/projected/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-kube-api-access-pf7c2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672998 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.672998 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.672953    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.674910 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.674888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.675035 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.675015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.681826 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.681808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7c2\" (UniqueName: \"kubernetes.io/projected/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-kube-api-access-pf7c2\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:49.804663 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:49.804583    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:50.095194 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:50.095118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"]
Sep 16 19:24:50.098553 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:24:50.098522    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f84720_b7eb_4da5_a37f_f9d983e1d79c.slice/crio-053dc701adeb293e15d2daaeb92c902282d9a4ea6c2fb7178a1f1cf48475db00 WatchSource:0}: Error finding container 053dc701adeb293e15d2daaeb92c902282d9a4ea6c2fb7178a1f1cf48475db00: Status 404 returned error can't find the container with id 053dc701adeb293e15d2daaeb92c902282d9a4ea6c2fb7178a1f1cf48475db00
Sep 16 19:24:50.883493 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:50.883452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" event={"ID":"b2f84720-b7eb-4da5-a37f-f9d983e1d79c","Type":"ContainerStarted","Data":"2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"}
Sep 16 19:24:50.883970 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:50.883534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" event={"ID":"b2f84720-b7eb-4da5-a37f-f9d983e1d79c","Type":"ContainerStarted","Data":"053dc701adeb293e15d2daaeb92c902282d9a4ea6c2fb7178a1f1cf48475db00"}
Sep 16 19:24:50.903453 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:50.903404    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" podStartSLOduration=1.9033899010000002 podStartE2EDuration="1.903389901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:50.901436192 +0000 UTC m=+2392.205453161" watchObservedRunningTime="2026-09-16 19:24:50.903389901 +0000 UTC m=+2392.207406868"
Sep 16 19:24:51.805771 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:51.805733    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:59.370116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.370082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:24:59.373334 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.373307    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:24:59.373937 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.373915    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:24:59.377171 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.377148    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:24:59.805378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.805345    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:59.817632 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.817608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:24:59.938081 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:24:59.938053    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:25:39.815490 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.815450    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"]
Sep 16 19:25:39.819186 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.819164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:39.822055 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.822034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:25:39.829543 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.829517    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"]
Sep 16 19:25:39.977595 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.977555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:39.977595 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.977598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:39.977828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.977658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:39.977828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.977678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:39.977828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.977736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87jm\" (UniqueName: \"kubernetes.io/projected/cfc640a5-cdd8-4429-a43f-113f03f7de5c-kube-api-access-q87jm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:39.977828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:39.977810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87jm\" (UniqueName: \"kubernetes.io/projected/cfc640a5-cdd8-4429-a43f-113f03f7de5c-kube-api-access-q87jm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.078935 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.079001 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.079001 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.078964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.080698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.080673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.080982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.080961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.089574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.089550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87jm\" (UniqueName: \"kubernetes.io/projected/cfc640a5-cdd8-4429-a43f-113f03f7de5c-kube-api-access-q87jm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.130814 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.130781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:40.404964 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:40.404881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"]
Sep 16 19:25:40.407128 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:25:40.407099    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc640a5_cdd8_4429_a43f_113f03f7de5c.slice/crio-d4c8fb318cd2b6b22451a0a5e6cf83ccd7fc69c01c3aceac1cf46d0b87bf8285 WatchSource:0}: Error finding container d4c8fb318cd2b6b22451a0a5e6cf83ccd7fc69c01c3aceac1cf46d0b87bf8285: Status 404 returned error can't find the container with id d4c8fb318cd2b6b22451a0a5e6cf83ccd7fc69c01c3aceac1cf46d0b87bf8285
Sep 16 19:25:41.134167 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:41.134131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" event={"ID":"cfc640a5-cdd8-4429-a43f-113f03f7de5c","Type":"ContainerStarted","Data":"8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"}
Sep 16 19:25:41.134571 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:41.134185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" event={"ID":"cfc640a5-cdd8-4429-a43f-113f03f7de5c","Type":"ContainerStarted","Data":"d4c8fb318cd2b6b22451a0a5e6cf83ccd7fc69c01c3aceac1cf46d0b87bf8285"}
Sep 16 19:25:41.152846 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:41.152796    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" podStartSLOduration=2.152782255 podStartE2EDuration="2.152782255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:41.150765232 +0000 UTC m=+2442.454782200" watchObservedRunningTime="2026-09-16 19:25:41.152782255 +0000 UTC m=+2442.456799222"
Sep 16 19:25:42.131536 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:42.131492    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:50.131606 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:50.131572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:50.146342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:50.146314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:25:50.188891 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:25:50.188839    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:26:22.927235 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:22.927198    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"]
Sep 16 19:26:22.936705 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:26:22.936660    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:26:22.936705 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:22.936705 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:22.936705 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89" cmd=["/bin/sleep","15"]
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:22.936729    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" message="" err=<
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:26:22.936745    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerName="main" containerID="cri-o://8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"
Sep 16 19:26:22.937029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:22.936799    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerName="main" containerID="cri-o://8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89" gracePeriod=60
Sep 16 19:26:23.178276 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.178193    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": dial tcp 10.132.0.69:8000: connect: connection refused"
Sep 16 19:26:23.288798 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.288774    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:26:23.340875 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.340819    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerID="8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89" exitCode=0
Sep 16 19:26:23.341005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.340889    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"
Sep 16 19:26:23.341005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.340896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" event={"ID":"cfc640a5-cdd8-4429-a43f-113f03f7de5c","Type":"ContainerDied","Data":"8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"}
Sep 16 19:26:23.341005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.340950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn" event={"ID":"cfc640a5-cdd8-4429-a43f-113f03f7de5c","Type":"ContainerDied","Data":"d4c8fb318cd2b6b22451a0a5e6cf83ccd7fc69c01c3aceac1cf46d0b87bf8285"}
Sep 16 19:26:23.341005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.340979    2581 scope.go:117] "RemoveContainer" containerID="8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"
Sep 16 19:26:23.344249 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344222    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tmp-dir\") pod \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") "
Sep 16 19:26:23.344377 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344278    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-home\") pod \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") "
Sep 16 19:26:23.344377 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-dshm\") pod \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") "
Sep 16 19:26:23.344503 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344418    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q87jm\" (UniqueName: \"kubernetes.io/projected/cfc640a5-cdd8-4429-a43f-113f03f7de5c-kube-api-access-q87jm\") pod \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") "
Sep 16 19:26:23.344503 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344481    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tls-certs\") pod \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") "
Sep 16 19:26:23.344611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344513    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cfc640a5-cdd8-4429-a43f-113f03f7de5c" (UID: "cfc640a5-cdd8-4429-a43f-113f03f7de5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:23.344611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-home" (OuterVolumeSpecName: "home") pod "cfc640a5-cdd8-4429-a43f-113f03f7de5c" (UID: "cfc640a5-cdd8-4429-a43f-113f03f7de5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:23.344611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-model-cache\") pod \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\" (UID: \"cfc640a5-cdd8-4429-a43f-113f03f7de5c\") "
Sep 16 19:26:23.344749 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "cfc640a5-cdd8-4429-a43f-113f03f7de5c" (UID: "cfc640a5-cdd8-4429-a43f-113f03f7de5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:23.344903 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344884    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.345008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344929    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.345008 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.344944    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.346784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.346760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-dshm" (OuterVolumeSpecName: "dshm") pod "cfc640a5-cdd8-4429-a43f-113f03f7de5c" (UID: "cfc640a5-cdd8-4429-a43f-113f03f7de5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:23.346912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.346812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cfc640a5-cdd8-4429-a43f-113f03f7de5c" (UID: "cfc640a5-cdd8-4429-a43f-113f03f7de5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:23.346912 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.346826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc640a5-cdd8-4429-a43f-113f03f7de5c-kube-api-access-q87jm" (OuterVolumeSpecName: "kube-api-access-q87jm") pod "cfc640a5-cdd8-4429-a43f-113f03f7de5c" (UID: "cfc640a5-cdd8-4429-a43f-113f03f7de5c"). InnerVolumeSpecName "kube-api-access-q87jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:23.350791 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.350762    2581 scope.go:117] "RemoveContainer" containerID="8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"
Sep 16 19:26:23.351044 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:26:23.351022    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89\": container with ID starting with 8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89 not found: ID does not exist" containerID="8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"
Sep 16 19:26:23.351112 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.351050    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89"} err="failed to get container status \"8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89\": rpc error: code = NotFound desc = could not find container \"8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89\": container with ID starting with 8b6e99b5a7f295bd56fcf58e75b7d0e825f70d3d9735b004a2a4a502b0b11f89 not found: ID does not exist"
Sep 16 19:26:23.446244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.446179    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfc640a5-cdd8-4429-a43f-113f03f7de5c-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.446244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.446202    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87jm\" (UniqueName: \"kubernetes.io/projected/cfc640a5-cdd8-4429-a43f-113f03f7de5c-kube-api-access-q87jm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.446244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.446212    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfc640a5-cdd8-4429-a43f-113f03f7de5c-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:26:23.666177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.666145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"]
Sep 16 19:26:23.670136 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:23.670109    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c9954s4jn"]
Sep 16 19:26:25.222386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:26:25.222349    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" path="/var/lib/kubelet/pods/cfc640a5-cdd8-4429-a43f-113f03f7de5c/volumes"
Sep 16 19:27:01.203559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:01.203527    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"]
Sep 16 19:27:13.875408 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:13.875373    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"]
Sep 16 19:27:13.885059 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:27:13.885023    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:27:13.885059 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:27:13.885059 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:27:13.885059 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab" cmd=["/bin/sleep","15"]
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:13.885077    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" message="" err=<
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:27:13.885093    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerName="main" containerID="cri-o://2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"
Sep 16 19:27:13.885365 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:13.885137    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerName="main" containerID="cri-o://2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab" gracePeriod=60
Sep 16 19:27:13.927814 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:13.927765    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 16 19:27:14.231709 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.231678    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:27:14.252279 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf7c2\" (UniqueName: \"kubernetes.io/projected/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-kube-api-access-pf7c2\") pod \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") "
Sep 16 19:27:14.252446 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-home\") pod \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") "
Sep 16 19:27:14.252446 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-model-cache\") pod \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") "
Sep 16 19:27:14.252446 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tmp-dir\") pod \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") "
Sep 16 19:27:14.252446 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252400    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tls-certs\") pod \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") "
Sep 16 19:27:14.252668 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-dshm\") pod \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\" (UID: \"b2f84720-b7eb-4da5-a37f-f9d983e1d79c\") "
Sep 16 19:27:14.252723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-home" (OuterVolumeSpecName: "home") pod "b2f84720-b7eb-4da5-a37f-f9d983e1d79c" (UID: "b2f84720-b7eb-4da5-a37f-f9d983e1d79c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:14.252723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252705    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2f84720-b7eb-4da5-a37f-f9d983e1d79c" (UID: "b2f84720-b7eb-4da5-a37f-f9d983e1d79c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:14.252837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252794    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.252837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.252822    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.253147 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.253124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2f84720-b7eb-4da5-a37f-f9d983e1d79c" (UID: "b2f84720-b7eb-4da5-a37f-f9d983e1d79c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:14.254714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.254691    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-kube-api-access-pf7c2" (OuterVolumeSpecName: "kube-api-access-pf7c2") pod "b2f84720-b7eb-4da5-a37f-f9d983e1d79c" (UID: "b2f84720-b7eb-4da5-a37f-f9d983e1d79c"). InnerVolumeSpecName "kube-api-access-pf7c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:14.254831 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.254816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2f84720-b7eb-4da5-a37f-f9d983e1d79c" (UID: "b2f84720-b7eb-4da5-a37f-f9d983e1d79c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:14.254914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.254890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-dshm" (OuterVolumeSpecName: "dshm") pod "b2f84720-b7eb-4da5-a37f-f9d983e1d79c" (UID: "b2f84720-b7eb-4da5-a37f-f9d983e1d79c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:14.354054 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.354012    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf7c2\" (UniqueName: \"kubernetes.io/projected/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-kube-api-access-pf7c2\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.354054 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.354050    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.354266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.354065    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.354266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.354078    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2f84720-b7eb-4da5-a37f-f9d983e1d79c-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:14.598781 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.598743    2581 generic.go:356] "Generic (PLEG): container finished" podID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerID="2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab" exitCode=0
Sep 16 19:27:14.598994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.598786    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"
Sep 16 19:27:14.598994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.598822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" event={"ID":"b2f84720-b7eb-4da5-a37f-f9d983e1d79c","Type":"ContainerDied","Data":"2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"}
Sep 16 19:27:14.598994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.598885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5" event={"ID":"b2f84720-b7eb-4da5-a37f-f9d983e1d79c","Type":"ContainerDied","Data":"053dc701adeb293e15d2daaeb92c902282d9a4ea6c2fb7178a1f1cf48475db00"}
Sep 16 19:27:14.598994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.598908    2581 scope.go:117] "RemoveContainer" containerID="2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"
Sep 16 19:27:14.608556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.608536    2581 scope.go:117] "RemoveContainer" containerID="2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"
Sep 16 19:27:14.608847 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:27:14.608825    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab\": container with ID starting with 2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab not found: ID does not exist" containerID="2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"
Sep 16 19:27:14.608958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.608868    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab"} err="failed to get container status \"2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab\": rpc error: code = NotFound desc = could not find container \"2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab\": container with ID starting with 2ec922089761a5588e132145d50bf8afdf55c024180125c4cb2b7c534d1b02ab not found: ID does not exist"
Sep 16 19:27:14.621670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.621643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"]
Sep 16 19:27:14.625345 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:14.625324    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-z4bw5"]
Sep 16 19:27:15.222872 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:15.222829    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" path="/var/lib/kubelet/pods/b2f84720-b7eb-4da5-a37f-f9d983e1d79c/volumes"
Sep 16 19:27:16.222349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.222311    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf" podUID="30e0600f-67bf-4d09-80fa-7a83095b63e4" containerName="main" containerID="cri-o://580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279" gracePeriod=45
Sep 16 19:27:16.482987 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.482934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:27:16.571934 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.571894    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-model-cache\") pod \"30e0600f-67bf-4d09-80fa-7a83095b63e4\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") "
Sep 16 19:27:16.572132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.571969    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-home\") pod \"30e0600f-67bf-4d09-80fa-7a83095b63e4\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") "
Sep 16 19:27:16.572132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e0600f-67bf-4d09-80fa-7a83095b63e4-tls-certs\") pod \"30e0600f-67bf-4d09-80fa-7a83095b63e4\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") "
Sep 16 19:27:16.572132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-tmp-dir\") pod \"30e0600f-67bf-4d09-80fa-7a83095b63e4\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") "
Sep 16 19:27:16.572132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk4zb\" (UniqueName: \"kubernetes.io/projected/30e0600f-67bf-4d09-80fa-7a83095b63e4-kube-api-access-rk4zb\") pod \"30e0600f-67bf-4d09-80fa-7a83095b63e4\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") "
Sep 16 19:27:16.572350 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-dshm\") pod \"30e0600f-67bf-4d09-80fa-7a83095b63e4\" (UID: \"30e0600f-67bf-4d09-80fa-7a83095b63e4\") "
Sep 16 19:27:16.572350 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "30e0600f-67bf-4d09-80fa-7a83095b63e4" (UID: "30e0600f-67bf-4d09-80fa-7a83095b63e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:16.572350 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-home" (OuterVolumeSpecName: "home") pod "30e0600f-67bf-4d09-80fa-7a83095b63e4" (UID: "30e0600f-67bf-4d09-80fa-7a83095b63e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:16.572511 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30e0600f-67bf-4d09-80fa-7a83095b63e4" (UID: "30e0600f-67bf-4d09-80fa-7a83095b63e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:16.572570 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572515    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:16.572570 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572534    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:16.572570 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.572548    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:16.574298 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.574272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-dshm" (OuterVolumeSpecName: "dshm") pod "30e0600f-67bf-4d09-80fa-7a83095b63e4" (UID: "30e0600f-67bf-4d09-80fa-7a83095b63e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:16.574403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.574310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e0600f-67bf-4d09-80fa-7a83095b63e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30e0600f-67bf-4d09-80fa-7a83095b63e4" (UID: "30e0600f-67bf-4d09-80fa-7a83095b63e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:16.574403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.574321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e0600f-67bf-4d09-80fa-7a83095b63e4-kube-api-access-rk4zb" (OuterVolumeSpecName: "kube-api-access-rk4zb") pod "30e0600f-67bf-4d09-80fa-7a83095b63e4" (UID: "30e0600f-67bf-4d09-80fa-7a83095b63e4"). InnerVolumeSpecName "kube-api-access-rk4zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:16.611656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.611621    2581 generic.go:356] "Generic (PLEG): container finished" podID="30e0600f-67bf-4d09-80fa-7a83095b63e4" containerID="580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279" exitCode=0
Sep 16 19:27:16.611833 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.611676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"
Sep 16 19:27:16.611833 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.611705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf" event={"ID":"30e0600f-67bf-4d09-80fa-7a83095b63e4","Type":"ContainerDied","Data":"580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279"}
Sep 16 19:27:16.611833 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.611755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf" event={"ID":"30e0600f-67bf-4d09-80fa-7a83095b63e4","Type":"ContainerDied","Data":"71f1575903ef7e416bfe6c841c9475b98ff755b38a1a77f8126db623f42a8dbe"}
Sep 16 19:27:16.611833 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.611776    2581 scope.go:117] "RemoveContainer" containerID="580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279"
Sep 16 19:27:16.620594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.620576    2581 scope.go:117] "RemoveContainer" containerID="580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279"
Sep 16 19:27:16.620887 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:27:16.620849    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279\": container with ID starting with 580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279 not found: ID does not exist" containerID="580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279"
Sep 16 19:27:16.620926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.620895    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279"} err="failed to get container status \"580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279\": rpc error: code = NotFound desc = could not find container \"580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279\": container with ID starting with 580729aca6709553f61a782a0798822a2f7df38c7ddf23489f4a58b49893f279 not found: ID does not exist"
Sep 16 19:27:16.634595 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.634576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"]
Sep 16 19:27:16.638565 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.638546    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-wkwvf"]
Sep 16 19:27:16.673939 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.673913    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e0600f-67bf-4d09-80fa-7a83095b63e4-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:16.673939 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.673936    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4zb\" (UniqueName: \"kubernetes.io/projected/30e0600f-67bf-4d09-80fa-7a83095b63e4-kube-api-access-rk4zb\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:16.674096 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:16.673945    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e0600f-67bf-4d09-80fa-7a83095b63e4-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:27:17.221596 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:27:17.221561    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e0600f-67bf-4d09-80fa-7a83095b63e4" path="/var/lib/kubelet/pods/30e0600f-67bf-4d09-80fa-7a83095b63e4/volumes"
Sep 16 19:28:11.372867 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.372830    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"]
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373294    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373313    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373342    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e0600f-67bf-4d09-80fa-7a83095b63e4" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373351    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e0600f-67bf-4d09-80fa-7a83095b63e4" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373380    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373457    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e0600f-67bf-4d09-80fa-7a83095b63e4" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373469    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f84720-b7eb-4da5-a37f-f9d983e1d79c" containerName="main"
Sep 16 19:28:11.373476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.373479    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc640a5-cdd8-4429-a43f-113f03f7de5c" containerName="main"
Sep 16 19:28:11.376419 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.376398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.379422 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.379392    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:28:11.379543 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.379461    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:28:11.379543 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.379501    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:28:11.379883 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.379865    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-r5s47\""
Sep 16 19:28:11.386970 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.386949    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"]
Sep 16 19:28:11.413965 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.413931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.414127 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.414010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.414127 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.414039    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-kube-api-access-vw7b7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.414127 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.414097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.414252 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.414166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.414252 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.414208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515362 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-kube-api-access-vw7b7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515362 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515362 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515362 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515584 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515706 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.515883 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.515845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.517637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.517608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.517729 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.517710    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.524902 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.524840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-kube-api-access-vw7b7\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.686429 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.686393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:11.958726 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:11.958695    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"]
Sep 16 19:28:11.960963 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:28:11.960937    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4b8fdb_a082_4a75_b6cf_2b4a9655a1a7.slice/crio-6fc37985ed281b1f4c260da240ca8cad0bed30762f53617eccd3bd2e765edea9 WatchSource:0}: Error finding container 6fc37985ed281b1f4c260da240ca8cad0bed30762f53617eccd3bd2e765edea9: Status 404 returned error can't find the container with id 6fc37985ed281b1f4c260da240ca8cad0bed30762f53617eccd3bd2e765edea9
Sep 16 19:28:12.889654 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:12.889616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" event={"ID":"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7","Type":"ContainerStarted","Data":"82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5"}
Sep 16 19:28:12.890095 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:12.889669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" event={"ID":"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7","Type":"ContainerStarted","Data":"6fc37985ed281b1f4c260da240ca8cad0bed30762f53617eccd3bd2e765edea9"}
Sep 16 19:28:12.909839 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:12.909785    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" podStartSLOduration=1.9097684560000001 podStartE2EDuration="1.909768456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:12.908166119 +0000 UTC m=+2594.212183086" watchObservedRunningTime="2026-09-16 19:28:12.909768456 +0000 UTC m=+2594.213785423"
Sep 16 19:28:13.687118 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:13.687076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:21.687227 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:21.687141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:21.699958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:21.699932    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:28:21.948719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:28:21.948626    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:29:46.411596 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.411564    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"]
Sep 16 19:29:46.414971 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.414953    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.417568 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.417549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:29:46.425531 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.425506    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"]
Sep 16 19:29:46.500821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.500789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.500821 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.500822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.501044 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.500848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e58c39-504c-4286-9677-9c5eafb4467d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.501044 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.500935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lsk\" (UniqueName: \"kubernetes.io/projected/e3e58c39-504c-4286-9677-9c5eafb4467d-kube-api-access-55lsk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.501044 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.500979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.501044 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.501007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602658 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e58c39-504c-4286-9677-9c5eafb4467d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602658 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lsk\" (UniqueName: \"kubernetes.io/projected/e3e58c39-504c-4286-9677-9c5eafb4467d-kube-api-access-55lsk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602658 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602658 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602883 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.602951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.603012 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.602995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.604811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.604790    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.605132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.605113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e58c39-504c-4286-9677-9c5eafb4467d-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.611184 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.611160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lsk\" (UniqueName: \"kubernetes.io/projected/e3e58c39-504c-4286-9677-9c5eafb4467d-kube-api-access-55lsk\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.662392 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.662323    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"]
Sep 16 19:29:46.666159 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.666140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.668707 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.668681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-cqvd9\""
Sep 16 19:29:46.675739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.675711    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"]
Sep 16 19:29:46.725911 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.725880    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:46.804308 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.804277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6ngt\" (UniqueName: \"kubernetes.io/projected/f22e10eb-9e8d-4bdd-88bc-c080d7895709-kube-api-access-v6ngt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.804435 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.804360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22e10eb-9e8d-4bdd-88bc-c080d7895709-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.905036 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.905000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6ngt\" (UniqueName: \"kubernetes.io/projected/f22e10eb-9e8d-4bdd-88bc-c080d7895709-kube-api-access-v6ngt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.905182 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.905082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22e10eb-9e8d-4bdd-88bc-c080d7895709-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.907450 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.907429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22e10eb-9e8d-4bdd-88bc-c080d7895709-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.914081 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.914028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6ngt\" (UniqueName: \"kubernetes.io/projected/f22e10eb-9e8d-4bdd-88bc-c080d7895709-kube-api-access-v6ngt\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.979449 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.979421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:46.998422 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:46.998398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"]
Sep 16 19:29:47.000018 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:29:46.999993    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e58c39_504c_4286_9677_9c5eafb4467d.slice/crio-3413ae23168b19e06f353aeb361b431291681ee230f3fc11b59ded77dd0d0f60 WatchSource:0}: Error finding container 3413ae23168b19e06f353aeb361b431291681ee230f3fc11b59ded77dd0d0f60: Status 404 returned error can't find the container with id 3413ae23168b19e06f353aeb361b431291681ee230f3fc11b59ded77dd0d0f60
Sep 16 19:29:47.253917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.253893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"]
Sep 16 19:29:47.255647 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:29:47.255623    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22e10eb_9e8d_4bdd_88bc_c080d7895709.slice/crio-4a4a2498cb1ae75107e8605fb35d50ca830722b1613585b19dec76f832badea0 WatchSource:0}: Error finding container 4a4a2498cb1ae75107e8605fb35d50ca830722b1613585b19dec76f832badea0: Status 404 returned error can't find the container with id 4a4a2498cb1ae75107e8605fb35d50ca830722b1613585b19dec76f832badea0
Sep 16 19:29:47.343792 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.343756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" event={"ID":"f22e10eb-9e8d-4bdd-88bc-c080d7895709","Type":"ContainerStarted","Data":"662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea"}
Sep 16 19:29:47.343966 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.343813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" event={"ID":"f22e10eb-9e8d-4bdd-88bc-c080d7895709","Type":"ContainerStarted","Data":"4a4a2498cb1ae75107e8605fb35d50ca830722b1613585b19dec76f832badea0"}
Sep 16 19:29:47.343966 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.343888    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:29:47.345671 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.345647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" event={"ID":"e3e58c39-504c-4286-9677-9c5eafb4467d","Type":"ContainerStarted","Data":"864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"}
Sep 16 19:29:47.345752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.345700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" event={"ID":"e3e58c39-504c-4286-9677-9c5eafb4467d","Type":"ContainerStarted","Data":"3413ae23168b19e06f353aeb361b431291681ee230f3fc11b59ded77dd0d0f60"}
Sep 16 19:29:47.360693 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.360485    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" podStartSLOduration=1.360470322 podStartE2EDuration="1.360470322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:47.359815228 +0000 UTC m=+2688.663832198" watchObservedRunningTime="2026-09-16 19:29:47.360470322 +0000 UTC m=+2688.664487304"
Sep 16 19:29:47.377035 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.376992    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" podStartSLOduration=1.376980434 podStartE2EDuration="1.376980434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:47.376422392 +0000 UTC m=+2688.680439360" watchObservedRunningTime="2026-09-16 19:29:47.376980434 +0000 UTC m=+2688.680997400"
Sep 16 19:29:47.726108 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:47.726074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:56.726603 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:56.726518    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:56.739093 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:56.739064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:57.406315 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:57.406278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:29:59.404103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:59.404067    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:29:59.407502 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:59.407476    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:29:59.409875 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:59.409837    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:29:59.412714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:29:59.412695    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:30:18.352130 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:18.352100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:30:20.256561 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.256525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"]
Sep 16 19:30:20.265907 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:30:20.265849    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:30:20.265907 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:20.265907 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:20.265907 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae" cmd=["/bin/sleep","15"]
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.265920    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" message="" err=<
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:30:20.265938    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerName="main" containerID="cri-o://864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"
Sep 16 19:30:20.266156 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.265976    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerName="main" containerID="cri-o://864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae" gracePeriod=60
Sep 16 19:30:20.279775 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.279743    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"]
Sep 16 19:30:20.396402 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.396365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Sep 16 19:30:20.510331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.510270    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:30:20.527242 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.527209    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerID="864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae" exitCode=0
Sep 16 19:30:20.527421 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.527249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"
Sep 16 19:30:20.527421 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.527263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" event={"ID":"e3e58c39-504c-4286-9677-9c5eafb4467d","Type":"ContainerDied","Data":"864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"}
Sep 16 19:30:20.527421 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.527314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98" event={"ID":"e3e58c39-504c-4286-9677-9c5eafb4467d","Type":"ContainerDied","Data":"3413ae23168b19e06f353aeb361b431291681ee230f3fc11b59ded77dd0d0f60"}
Sep 16 19:30:20.527421 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.527344    2581 scope.go:117] "RemoveContainer" containerID="864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"
Sep 16 19:30:20.537613 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.537593    2581 scope.go:117] "RemoveContainer" containerID="864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"
Sep 16 19:30:20.538062 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:30:20.538036    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae\": container with ID starting with 864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae not found: ID does not exist" containerID="864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"
Sep 16 19:30:20.538136 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.538070    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae"} err="failed to get container status \"864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae\": rpc error: code = NotFound desc = could not find container \"864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae\": container with ID starting with 864f174a27b5f9a89903162958114cf601962cf73b7d06f2bcdfa165ad5f4dae not found: ID does not exist"
Sep 16 19:30:20.575423 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-tmp-dir\") pod \"e3e58c39-504c-4286-9677-9c5eafb4467d\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") "
Sep 16 19:30:20.575623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-dshm\") pod \"e3e58c39-504c-4286-9677-9c5eafb4467d\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") "
Sep 16 19:30:20.575623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575473    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e58c39-504c-4286-9677-9c5eafb4467d-tls-certs\") pod \"e3e58c39-504c-4286-9677-9c5eafb4467d\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") "
Sep 16 19:30:20.575623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-model-cache\") pod \"e3e58c39-504c-4286-9677-9c5eafb4467d\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") "
Sep 16 19:30:20.575623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-home\") pod \"e3e58c39-504c-4286-9677-9c5eafb4467d\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") "
Sep 16 19:30:20.575623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55lsk\" (UniqueName: \"kubernetes.io/projected/e3e58c39-504c-4286-9677-9c5eafb4467d-kube-api-access-55lsk\") pod \"e3e58c39-504c-4286-9677-9c5eafb4467d\" (UID: \"e3e58c39-504c-4286-9677-9c5eafb4467d\") "
Sep 16 19:30:20.575942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575692    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3e58c39-504c-4286-9677-9c5eafb4467d" (UID: "e3e58c39-504c-4286-9677-9c5eafb4467d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.575942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3e58c39-504c-4286-9677-9c5eafb4467d" (UID: "e3e58c39-504c-4286-9677-9c5eafb4467d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.575942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575921    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.575942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-home" (OuterVolumeSpecName: "home") pod "e3e58c39-504c-4286-9677-9c5eafb4467d" (UID: "e3e58c39-504c-4286-9677-9c5eafb4467d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.575942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.575941    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.577796 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.577769    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-dshm" (OuterVolumeSpecName: "dshm") pod "e3e58c39-504c-4286-9677-9c5eafb4467d" (UID: "e3e58c39-504c-4286-9677-9c5eafb4467d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:30:20.577916 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.577806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e58c39-504c-4286-9677-9c5eafb4467d-kube-api-access-55lsk" (OuterVolumeSpecName: "kube-api-access-55lsk") pod "e3e58c39-504c-4286-9677-9c5eafb4467d" (UID: "e3e58c39-504c-4286-9677-9c5eafb4467d"). InnerVolumeSpecName "kube-api-access-55lsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:30:20.577916 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.577811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e58c39-504c-4286-9677-9c5eafb4467d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3e58c39-504c-4286-9677-9c5eafb4467d" (UID: "e3e58c39-504c-4286-9677-9c5eafb4467d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:30:20.676666 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.676627    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.676666 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.676666    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e58c39-504c-4286-9677-9c5eafb4467d-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.676882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.676684    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e58c39-504c-4286-9677-9c5eafb4467d-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.676882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.676698    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55lsk\" (UniqueName: \"kubernetes.io/projected/e3e58c39-504c-4286-9677-9c5eafb4467d-kube-api-access-55lsk\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:30:20.852438 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.852404    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"]
Sep 16 19:30:20.863122 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:20.863091    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-46f98"]
Sep 16 19:30:21.223765 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:21.223720    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" path="/var/lib/kubelet/pods/e3e58c39-504c-4286-9677-9c5eafb4467d/volumes"
Sep 16 19:30:28.352119 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:28.352072    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:35.298181 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:35.298119    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main" containerID="cri-o://662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea" gracePeriod=45
Sep 16 19:30:38.352671 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:38.352632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:48.353296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:48.353257    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:48.353829 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:48.353384    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:30:58.352276 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:30:58.352236    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:05.543528 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.543502    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:31:05.554295 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.554267    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6ngt\" (UniqueName: \"kubernetes.io/projected/f22e10eb-9e8d-4bdd-88bc-c080d7895709-kube-api-access-v6ngt\") pod \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") "
Sep 16 19:31:05.554413 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.554311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22e10eb-9e8d-4bdd-88bc-c080d7895709-tls-certs\") pod \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\" (UID: \"f22e10eb-9e8d-4bdd-88bc-c080d7895709\") "
Sep 16 19:31:05.556318 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.556281    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22e10eb-9e8d-4bdd-88bc-c080d7895709-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f22e10eb-9e8d-4bdd-88bc-c080d7895709" (UID: "f22e10eb-9e8d-4bdd-88bc-c080d7895709"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:05.556422 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.556341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22e10eb-9e8d-4bdd-88bc-c080d7895709-kube-api-access-v6ngt" (OuterVolumeSpecName: "kube-api-access-v6ngt") pod "f22e10eb-9e8d-4bdd-88bc-c080d7895709" (UID: "f22e10eb-9e8d-4bdd-88bc-c080d7895709"). InnerVolumeSpecName "kube-api-access-v6ngt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:05.655397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.655366    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f22e10eb-9e8d-4bdd-88bc-c080d7895709-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:05.655397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.655392    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6ngt\" (UniqueName: \"kubernetes.io/projected/f22e10eb-9e8d-4bdd-88bc-c080d7895709-kube-api-access-v6ngt\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:05.747064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.747028    2581 generic.go:356] "Generic (PLEG): container finished" podID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerID="662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea" exitCode=0
Sep 16 19:31:05.747217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.747082    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"
Sep 16 19:31:05.747217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.747108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" event={"ID":"f22e10eb-9e8d-4bdd-88bc-c080d7895709","Type":"ContainerDied","Data":"662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea"}
Sep 16 19:31:05.747217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.747167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk" event={"ID":"f22e10eb-9e8d-4bdd-88bc-c080d7895709","Type":"ContainerDied","Data":"4a4a2498cb1ae75107e8605fb35d50ca830722b1613585b19dec76f832badea0"}
Sep 16 19:31:05.747217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.747190    2581 scope.go:117] "RemoveContainer" containerID="662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea"
Sep 16 19:31:05.756119 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.756101    2581 scope.go:117] "RemoveContainer" containerID="662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea"
Sep 16 19:31:05.756393 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:31:05.756376    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea\": container with ID starting with 662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea not found: ID does not exist" containerID="662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea"
Sep 16 19:31:05.756440 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.756402    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea"} err="failed to get container status \"662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea\": rpc error: code = NotFound desc = could not find container \"662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea\": container with ID starting with 662aecc3f4c8a17914e5b319a338f695f4285bdc3d28cfce05068a90df5991ea not found: ID does not exist"
Sep 16 19:31:05.770788 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.770763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"]
Sep 16 19:31:05.777081 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:05.777060    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche7jkwk"]
Sep 16 19:31:07.222229 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:07.222199    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" path="/var/lib/kubelet/pods/f22e10eb-9e8d-4bdd-88bc-c080d7895709/volumes"
Sep 16 19:31:12.513709 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.513672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"]
Sep 16 19:31:12.523273 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:31:12.523232    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:31:12.523273 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:31:12.523273 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:31:12.523273 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5" cmd=["/bin/sleep","15"]
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.523291    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" message="" err=<
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:31:12.523303    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" podUID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" containerName="main" containerID="cri-o://82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5"
Sep 16 19:31:12.523585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.523352    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" podUID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" containerName="main" containerID="cri-o://82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5" gracePeriod=60
Sep 16 19:31:12.782087 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.782008    2581 generic.go:356] "Generic (PLEG): container finished" podID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" containerID="82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5" exitCode=0
Sep 16 19:31:12.782231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.782089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" event={"ID":"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7","Type":"ContainerDied","Data":"82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5"}
Sep 16 19:31:12.871469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.871442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:31:12.908234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908204    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-home\") pod \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") "
Sep 16 19:31:12.908358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908244    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-kube-api-access-vw7b7\") pod \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") "
Sep 16 19:31:12.908358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908267    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tmp-dir\") pod \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") "
Sep 16 19:31:12.908358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-model-cache\") pod \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") "
Sep 16 19:31:12.908358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tls-certs\") pod \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") "
Sep 16 19:31:12.908358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-dshm\") pod \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\" (UID: \"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7\") "
Sep 16 19:31:12.908621 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" (UID: "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:12.908621 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908543    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-home" (OuterVolumeSpecName: "home") pod "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" (UID: "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:12.908735 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908618    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" (UID: "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:12.908914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908891    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:12.909007 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908918    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:12.909007 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.908934    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:12.910290 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.910262    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-kube-api-access-vw7b7" (OuterVolumeSpecName: "kube-api-access-vw7b7") pod "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" (UID: "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7"). InnerVolumeSpecName "kube-api-access-vw7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:12.910387 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.910314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" (UID: "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:12.910431 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:12.910394    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-dshm" (OuterVolumeSpecName: "dshm") pod "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" (UID: "7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:13.009285 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.009251    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:13.009285 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.009279    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:13.009455 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.009292    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw7b7\" (UniqueName: \"kubernetes.io/projected/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7-kube-api-access-vw7b7\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:31:13.788956 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.788926    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"
Sep 16 19:31:13.789379 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.788920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv" event={"ID":"7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7","Type":"ContainerDied","Data":"6fc37985ed281b1f4c260da240ca8cad0bed30762f53617eccd3bd2e765edea9"}
Sep 16 19:31:13.789379 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.789063    2581 scope.go:117] "RemoveContainer" containerID="82e7d8618d4602ecd5c5aa039db45922058ad97504af22c7abb4c643badb07a5"
Sep 16 19:31:13.808322 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.808296    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"]
Sep 16 19:31:13.813892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:13.813867    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969qtdqv"]
Sep 16 19:31:15.221919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:31:15.221885    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" path="/var/lib/kubelet/pods/7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7/volumes"
Sep 16 19:32:24.728113 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728064    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"]
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728512    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728540    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728549    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728566    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728572    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728623    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e58c39-504c-4286-9677-9c5eafb4467d" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728632    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22e10eb-9e8d-4bdd-88bc-c080d7895709" containerName="main"
Sep 16 19:32:24.728641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.728637    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b4b8fdb-a082-4a75-b6cf-2b4a9655a1a7" containerName="main"
Sep 16 19:32:24.731656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.731636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.738063 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.738033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-dzg4f\""
Sep 16 19:32:24.738227 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.738207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:32:24.738558 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.738540    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 19:32:24.738940 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.738923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:32:24.739098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.739071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:32:24.739177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.739127    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:32:24.744266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.744243    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"]
Sep 16 19:32:24.769627 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769762 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4ps\" (UniqueName: \"kubernetes.io/projected/4220fcc1-ce18-4337-96c0-0f015172154c-kube-api-access-hb4ps\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769762 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4220fcc1-ce18-4337-96c0-0f015172154c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769762 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4220fcc1-ce18-4337-96c0-0f015172154c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.769951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.769877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871104 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871104 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4ps\" (UniqueName: \"kubernetes.io/projected/4220fcc1-ce18-4337-96c0-0f015172154c-kube-api-access-hb4ps\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4220fcc1-ce18-4337-96c0-0f015172154c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4220fcc1-ce18-4337-96c0-0f015172154c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871691 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871691 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.871978 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.871955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4220fcc1-ce18-4337-96c0-0f015172154c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.873496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.873474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.873760 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.873743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4220fcc1-ce18-4337-96c0-0f015172154c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:24.880117 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:24.880089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4ps\" (UniqueName: \"kubernetes.io/projected/4220fcc1-ce18-4337-96c0-0f015172154c-kube-api-access-hb4ps\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:25.049496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:25.049409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:25.335427 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:25.335396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"]
Sep 16 19:32:25.337267 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:32:25.337241    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4220fcc1_ce18_4337_96c0_0f015172154c.slice/crio-b9298ffb6c0803b93bff271e05b5c46a14f7485ce2011f3c82102f4a4c79a317 WatchSource:0}: Error finding container b9298ffb6c0803b93bff271e05b5c46a14f7485ce2011f3c82102f4a4c79a317: Status 404 returned error can't find the container with id b9298ffb6c0803b93bff271e05b5c46a14f7485ce2011f3c82102f4a4c79a317
Sep 16 19:32:26.136409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:26.136373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" event={"ID":"4220fcc1-ce18-4337-96c0-0f015172154c","Type":"ContainerStarted","Data":"bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0"}
Sep 16 19:32:26.136790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:26.136428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" event={"ID":"4220fcc1-ce18-4337-96c0-0f015172154c","Type":"ContainerStarted","Data":"b9298ffb6c0803b93bff271e05b5c46a14f7485ce2011f3c82102f4a4c79a317"}
Sep 16 19:32:29.156158 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:29.156120    2581 generic.go:356] "Generic (PLEG): container finished" podID="4220fcc1-ce18-4337-96c0-0f015172154c" containerID="bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0" exitCode=0
Sep 16 19:32:29.156601 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:29.156192    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" event={"ID":"4220fcc1-ce18-4337-96c0-0f015172154c","Type":"ContainerDied","Data":"bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0"}
Sep 16 19:32:30.163395 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:30.163352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" event={"ID":"4220fcc1-ce18-4337-96c0-0f015172154c","Type":"ContainerStarted","Data":"03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d"}
Sep 16 19:32:30.188355 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:30.188265    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podStartSLOduration=3.347700922 podStartE2EDuration="6.188243618s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.840542696s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:32:30.18621005 +0000 UTC m=+2851.490227018" watchObservedRunningTime="2026-09-16 19:32:30.188243618 +0000 UTC m=+2851.492260587"
Sep 16 19:32:31.049769 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:31.049731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:35.050020 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:35.049975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:32:35.051643 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:35.051612    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:32:45.050472 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:45.050424    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:32:55.050385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:32:55.050291    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:33:05.050072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:33:05.050026    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:33:15.050304 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:33:15.050257    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:33:25.050689 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:33:25.050634    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:33:35.050452 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:33:35.050402    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Sep 16 19:33:45.060086 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:33:45.060037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:33:45.067634 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:33:45.067607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:34:09.045756 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:09.045724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"]
Sep 16 19:34:12.072811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.072779    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"]
Sep 16 19:34:12.077274 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.077250    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.080345 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.080325    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-7mbdt\""
Sep 16 19:34:12.080471 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.080317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 16 19:34:12.080673 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.080648    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 16 19:34:12.081403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.081384    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 16 19:34:12.099314 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.099288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"]
Sep 16 19:34:12.213254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdwz\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-kube-api-access-hxdwz\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9336b085-f79b-4e0f-ab70-afe25e1634d0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.213594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.213511    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.314666 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.314628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.314845 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.314689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.314845 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.314804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.314971 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.314853    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.314971 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.314912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdwz\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-kube-api-access-hxdwz\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.314971 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.314942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9336b085-f79b-4e0f-ab70-afe25e1634d0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315389 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-certs\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-credential-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315669 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-data\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315669 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-socket\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.315909 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.315891    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9336b085-f79b-4e0f-ab70-afe25e1634d0-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.317072 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.317053    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-envoy\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.317479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.317459    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-podinfo\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.326930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.326871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdwz\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-kube-api-access-hxdwz\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.328048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.328026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-token\") pod \"router-gateway-1-openshift-default-5bb8f7bc87-hz88k\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.389196 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.389160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:12.683505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.683475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"]
Sep 16 19:34:12.684746 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:34:12.684721    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9336b085_f79b_4e0f_ab70_afe25e1634d0.slice/crio-ede57aba2ef38c849e9d6d280d00e4510057e63f0f6f90f68679afc7c952673b WatchSource:0}: Error finding container ede57aba2ef38c849e9d6d280d00e4510057e63f0f6f90f68679afc7c952673b: Status 404 returned error can't find the container with id ede57aba2ef38c849e9d6d280d00e4510057e63f0f6f90f68679afc7c952673b
Sep 16 19:34:12.686789 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.686753    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:34:12.686888 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.686819    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:34:12.686888 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.686849    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:34:12.708909 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:12.708881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" event={"ID":"9336b085-f79b-4e0f-ab70-afe25e1634d0","Type":"ContainerStarted","Data":"ede57aba2ef38c849e9d6d280d00e4510057e63f0f6f90f68679afc7c952673b"}
Sep 16 19:34:13.715529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:13.715488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" event={"ID":"9336b085-f79b-4e0f-ab70-afe25e1634d0","Type":"ContainerStarted","Data":"85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"}
Sep 16 19:34:13.737056 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:13.737004    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podStartSLOduration=1.7369876450000001 podStartE2EDuration="1.736987645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:13.735529826 +0000 UTC m=+2955.039546794" watchObservedRunningTime="2026-09-16 19:34:13.736987645 +0000 UTC m=+2955.041004612"
Sep 16 19:34:14.390042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:14.389999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:14.394946 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:14.394922    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:14.720111 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:14.720079    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:14.721074 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:14.721051    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:24.068321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.068229    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main" containerID="cri-o://03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d" gracePeriod=45
Sep 16 19:34:24.713125 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.713101    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:34:24.771589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.771517    2581 generic.go:356] "Generic (PLEG): container finished" podID="4220fcc1-ce18-4337-96c0-0f015172154c" containerID="03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d" exitCode=0
Sep 16 19:34:24.771589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.771576    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"
Sep 16 19:34:24.771741 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.771594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" event={"ID":"4220fcc1-ce18-4337-96c0-0f015172154c","Type":"ContainerDied","Data":"03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d"}
Sep 16 19:34:24.771741 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.771644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn" event={"ID":"4220fcc1-ce18-4337-96c0-0f015172154c","Type":"ContainerDied","Data":"b9298ffb6c0803b93bff271e05b5c46a14f7485ce2011f3c82102f4a4c79a317"}
Sep 16 19:34:24.771741 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.771667    2581 scope.go:117] "RemoveContainer" containerID="03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d"
Sep 16 19:34:24.782759 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.782742    2581 scope.go:117] "RemoveContainer" containerID="bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0"
Sep 16 19:34:24.790588 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.790571    2581 scope.go:117] "RemoveContainer" containerID="03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d"
Sep 16 19:34:24.790838 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:34:24.790819    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d\": container with ID starting with 03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d not found: ID does not exist" containerID="03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d"
Sep 16 19:34:24.790927 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.790846    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d"} err="failed to get container status \"03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d\": rpc error: code = NotFound desc = could not find container \"03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d\": container with ID starting with 03c5994e72bdad768f4da7e8f017418e59b7525ecf08a6c81583ee6319065b8d not found: ID does not exist"
Sep 16 19:34:24.790927 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.790882    2581 scope.go:117] "RemoveContainer" containerID="bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0"
Sep 16 19:34:24.791151 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:34:24.791134    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0\": container with ID starting with bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0 not found: ID does not exist" containerID="bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0"
Sep 16 19:34:24.791192 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.791165    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0"} err="failed to get container status \"bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0\": rpc error: code = NotFound desc = could not find container \"bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0\": container with ID starting with bb6026c1c32a3390a7d5607d60325de66017d2f76cd2a6014c52cf322a444ee0 not found: ID does not exist"
Sep 16 19:34:24.822421 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-home\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822499 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4220fcc1-ce18-4337-96c0-0f015172154c-cabundle-cert\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822499 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4220fcc1-ce18-4337-96c0-0f015172154c-tls-certs\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822499 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-kserve-provision-location\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822665    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-model-cache\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-dshm\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb4ps\" (UniqueName: \"kubernetes.io/projected/4220fcc1-ce18-4337-96c0-0f015172154c-kube-api-access-hb4ps\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.822816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822781    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-home" (OuterVolumeSpecName: "home") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:24.822816 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-tmp-dir\") pod \"4220fcc1-ce18-4337-96c0-0f015172154c\" (UID: \"4220fcc1-ce18-4337-96c0-0f015172154c\") "
Sep 16 19:34:24.823060 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822822    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4220fcc1-ce18-4337-96c0-0f015172154c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:24.823060 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.822951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:24.823168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.823122    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.823168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.823142    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4220fcc1-ce18-4337-96c0-0f015172154c-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.823168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.823158    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.824933 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.824903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-dshm" (OuterVolumeSpecName: "dshm") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:24.825084 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.825068    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4220fcc1-ce18-4337-96c0-0f015172154c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:24.825140 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.825104    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4220fcc1-ce18-4337-96c0-0f015172154c-kube-api-access-hb4ps" (OuterVolumeSpecName: "kube-api-access-hb4ps") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "kube-api-access-hb4ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:24.834996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.834971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:24.845064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.845043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4220fcc1-ce18-4337-96c0-0f015172154c" (UID: "4220fcc1-ce18-4337-96c0-0f015172154c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:24.924206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.924180    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.924206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.924203    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.924342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.924217    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4ps\" (UniqueName: \"kubernetes.io/projected/4220fcc1-ce18-4337-96c0-0f015172154c-kube-api-access-hb4ps\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.924342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.924230    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4220fcc1-ce18-4337-96c0-0f015172154c-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:24.924342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:24.924242    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4220fcc1-ce18-4337-96c0-0f015172154c-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:34:25.096834 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.096804    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"]
Sep 16 19:34:25.103892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.102014    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-7c5445f47f-778mn"]
Sep 16 19:34:25.222259 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.222225    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" path="/var/lib/kubelet/pods/4220fcc1-ce18-4337-96c0-0f015172154c/volumes"
Sep 16 19:34:25.500880 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.500837    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"]
Sep 16 19:34:25.501183 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.501171    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main"
Sep 16 19:34:25.501234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.501185    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main"
Sep 16 19:34:25.501234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.501207    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="storage-initializer"
Sep 16 19:34:25.501234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.501212    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="storage-initializer"
Sep 16 19:34:25.501335 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.501258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4220fcc1-ce18-4337-96c0-0f015172154c" containerName="main"
Sep 16 19:34:25.505607 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.505586    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.509000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.508978    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-mv8sq\""
Sep 16 19:34:25.509110 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.509001    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 16 19:34:25.515876 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.515839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"]
Sep 16 19:34:25.630280 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.630249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a633113-ca8c-42f6-8291-e4953f194490-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.630468 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.630292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.630468 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.630312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.630468 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.630381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.630468 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.630413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt6p\" (UniqueName: \"kubernetes.io/projected/1a633113-ca8c-42f6-8291-e4953f194490-kube-api-access-2nt6p\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.630468 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.630435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731379 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt6p\" (UniqueName: \"kubernetes.io/projected/1a633113-ca8c-42f6-8291-e4953f194490-kube-api-access-2nt6p\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a633113-ca8c-42f6-8291-e4953f194490-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731804 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731901 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.731901 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.732017 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.731961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.733781 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.733759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.734044 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.734029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a633113-ca8c-42f6-8291-e4953f194490-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.741150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.741119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt6p\" (UniqueName: \"kubernetes.io/projected/1a633113-ca8c-42f6-8291-e4953f194490-kube-api-access-2nt6p\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:25.816306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:25.816225    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:26.089554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:26.089529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"]
Sep 16 19:34:26.091400 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:34:26.091371    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a633113_ca8c_42f6_8291_e4953f194490.slice/crio-6e8a5a5fc1115f3c245bf9db1fdf97ecd644bedea8574d22cecbfeec5e534bce WatchSource:0}: Error finding container 6e8a5a5fc1115f3c245bf9db1fdf97ecd644bedea8574d22cecbfeec5e534bce: Status 404 returned error can't find the container with id 6e8a5a5fc1115f3c245bf9db1fdf97ecd644bedea8574d22cecbfeec5e534bce
Sep 16 19:34:26.783183 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:26.783142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" event={"ID":"1a633113-ca8c-42f6-8291-e4953f194490","Type":"ContainerStarted","Data":"2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"}
Sep 16 19:34:26.783683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:26.783211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" event={"ID":"1a633113-ca8c-42f6-8291-e4953f194490","Type":"ContainerStarted","Data":"6e8a5a5fc1115f3c245bf9db1fdf97ecd644bedea8574d22cecbfeec5e534bce"}
Sep 16 19:34:26.802092 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:26.802045    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" podStartSLOduration=1.802030765 podStartE2EDuration="1.802030765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:34:26.801171893 +0000 UTC m=+2968.105188860" watchObservedRunningTime="2026-09-16 19:34:26.802030765 +0000 UTC m=+2968.106047731"
Sep 16 19:34:26.816719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:26.816694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:35.817412 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.817381    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:35.829746 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.829722    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:35.830680 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.830653    2581 generic.go:356] "Generic (PLEG): container finished" podID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerID="85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1" exitCode=0
Sep 16 19:34:35.830801 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.830723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" event={"ID":"9336b085-f79b-4e0f-ab70-afe25e1634d0","Type":"ContainerDied","Data":"85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"}
Sep 16 19:34:35.831196 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.831180    2581 scope.go:117] "RemoveContainer" containerID="85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"
Sep 16 19:34:35.832105 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.832078    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:34:35.832167 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.832142    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:34:35.832204 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.832173    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 19:34:35.842760 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:35.842739    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:34:36.838259 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:36.838222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" event={"ID":"9336b085-f79b-4e0f-ab70-afe25e1634d0","Type":"ContainerStarted","Data":"7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1"}
Sep 16 19:34:37.390115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:37.390074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:37.390306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:37.390262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 19:34:38.389590 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:38.389545    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 19:34:39.389785 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:39.389745    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 19:34:40.390000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:40.389960    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 19:34:41.390097 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:41.390048    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 16 19:34:42.394169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:42.394137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:42.394640 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:42.394517    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:42.395228 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:42.395208    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:34:59.438335 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:59.438307    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:34:59.441756 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:59.441732    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:34:59.445424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:59.445399    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:34:59.448636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:34:59.448618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:35:44.449031 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.448992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"]
Sep 16 19:35:44.457679 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:35:44.457640    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:35:44.457679 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:35:44.457679 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:35:44.457679 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c" cmd=["/bin/sleep","15"]
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.457689    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" message="" err=<
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:35:44.457702    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" podUID="1a633113-ca8c-42f6-8291-e4953f194490" containerName="main" containerID="cri-o://2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"
Sep 16 19:35:44.457958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.457744    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" podUID="1a633113-ca8c-42f6-8291-e4953f194490" containerName="main" containerID="cri-o://2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c" gracePeriod=60
Sep 16 19:35:44.811091 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.811065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:35:44.986380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-tmp-dir\") pod \"1a633113-ca8c-42f6-8291-e4953f194490\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") "
Sep 16 19:35:44.986380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-model-cache\") pod \"1a633113-ca8c-42f6-8291-e4953f194490\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") "
Sep 16 19:35:44.986380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-dshm\") pod \"1a633113-ca8c-42f6-8291-e4953f194490\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") "
Sep 16 19:35:44.986670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-home\") pod \"1a633113-ca8c-42f6-8291-e4953f194490\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") "
Sep 16 19:35:44.986670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nt6p\" (UniqueName: \"kubernetes.io/projected/1a633113-ca8c-42f6-8291-e4953f194490-kube-api-access-2nt6p\") pod \"1a633113-ca8c-42f6-8291-e4953f194490\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") "
Sep 16 19:35:44.986670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986497    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a633113-ca8c-42f6-8291-e4953f194490-tls-certs\") pod \"1a633113-ca8c-42f6-8291-e4953f194490\" (UID: \"1a633113-ca8c-42f6-8291-e4953f194490\") "
Sep 16 19:35:44.986670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986584    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a633113-ca8c-42f6-8291-e4953f194490" (UID: "1a633113-ca8c-42f6-8291-e4953f194490"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:44.986670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986628    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-home" (OuterVolumeSpecName: "home") pod "1a633113-ca8c-42f6-8291-e4953f194490" (UID: "1a633113-ca8c-42f6-8291-e4953f194490"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:44.986885 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a633113-ca8c-42f6-8291-e4953f194490" (UID: "1a633113-ca8c-42f6-8291-e4953f194490"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:44.986885 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986734    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:44.986885 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986752    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:44.986885 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.986764    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:44.988631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.988608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-dshm" (OuterVolumeSpecName: "dshm") pod "1a633113-ca8c-42f6-8291-e4953f194490" (UID: "1a633113-ca8c-42f6-8291-e4953f194490"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:44.988738 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.988698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a633113-ca8c-42f6-8291-e4953f194490-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a633113-ca8c-42f6-8291-e4953f194490" (UID: "1a633113-ca8c-42f6-8291-e4953f194490"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:44.988738 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:44.988710    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a633113-ca8c-42f6-8291-e4953f194490-kube-api-access-2nt6p" (OuterVolumeSpecName: "kube-api-access-2nt6p") pod "1a633113-ca8c-42f6-8291-e4953f194490" (UID: "1a633113-ca8c-42f6-8291-e4953f194490"). InnerVolumeSpecName "kube-api-access-2nt6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:45.087517 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.087486    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt6p\" (UniqueName: \"kubernetes.io/projected/1a633113-ca8c-42f6-8291-e4953f194490-kube-api-access-2nt6p\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:45.087517 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.087512    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a633113-ca8c-42f6-8291-e4953f194490-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:45.087517 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.087522    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a633113-ca8c-42f6-8291-e4953f194490-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:45.193747 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.193714    2581 generic.go:356] "Generic (PLEG): container finished" podID="1a633113-ca8c-42f6-8291-e4953f194490" containerID="2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c" exitCode=0
Sep 16 19:35:45.193921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.193753    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"
Sep 16 19:35:45.193921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.193768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" event={"ID":"1a633113-ca8c-42f6-8291-e4953f194490","Type":"ContainerDied","Data":"2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"}
Sep 16 19:35:45.193921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.193813    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln" event={"ID":"1a633113-ca8c-42f6-8291-e4953f194490","Type":"ContainerDied","Data":"6e8a5a5fc1115f3c245bf9db1fdf97ecd644bedea8574d22cecbfeec5e534bce"}
Sep 16 19:35:45.193921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.193834    2581 scope.go:117] "RemoveContainer" containerID="2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"
Sep 16 19:35:45.203037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.203013    2581 scope.go:117] "RemoveContainer" containerID="2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"
Sep 16 19:35:45.203292 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:35:45.203270    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c\": container with ID starting with 2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c not found: ID does not exist" containerID="2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"
Sep 16 19:35:45.203372 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.203302    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c"} err="failed to get container status \"2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c\": rpc error: code = NotFound desc = could not find container \"2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c\": container with ID starting with 2fca2b549e642f5695c9d4f17e9a29bd9246d9e021a024ed8d519b4dd066f83c not found: ID does not exist"
Sep 16 19:35:45.215471 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.215449    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"]
Sep 16 19:35:45.221652 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:45.221632    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997ll7ln"]
Sep 16 19:35:47.221296 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:47.221261    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a633113-ca8c-42f6-8291-e4953f194490" path="/var/lib/kubelet/pods/1a633113-ca8c-42f6-8291-e4953f194490/volumes"
Sep 16 19:35:54.429780 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:54.429678    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"]
Sep 16 19:35:54.430335 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:54.429975    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" containerID="cri-o://7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1" gracePeriod=30
Sep 16 19:35:57.397076 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:57.397031    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:35:59.683265 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.683238    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:35:59.699169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699103    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-credential-socket\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699143    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-token\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxdwz\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-kube-api-access-hxdwz\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699347 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699191    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-socket\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699414 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:59.699462 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699388    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-podinfo\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699462 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699421    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:59.699550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-envoy\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9336b085-f79b-4e0f-ab70-afe25e1634d0-istiod-ca-cert\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-certs\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699661 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699592    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-data\") pod \"9336b085-f79b-4e0f-ab70-afe25e1634d0\" (UID: \"9336b085-f79b-4e0f-ab70-afe25e1634d0\") "
Sep 16 19:35:59.699845 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699828    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-credential-socket\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.699951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699843    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:59.699951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699852    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-socket\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.699951 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:59.700118 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.699969    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9336b085-f79b-4e0f-ab70-afe25e1634d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:59.701463 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.701435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 19:35:59.701567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.701470    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:59.701567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.701558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:59.701644 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.701607    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-kube-api-access-hxdwz" (OuterVolumeSpecName: "kube-api-access-hxdwz") pod "9336b085-f79b-4e0f-ab70-afe25e1634d0" (UID: "9336b085-f79b-4e0f-ab70-afe25e1634d0"). InnerVolumeSpecName "kube-api-access-hxdwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:59.801050 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801016    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdwz\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-kube-api-access-hxdwz\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.801050 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801043    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-podinfo\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.801050 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801052    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-envoy\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.801273 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801061    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9336b085-f79b-4e0f-ab70-afe25e1634d0-istiod-ca-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.801273 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801069    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-workload-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.801273 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801077    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-data\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:35:59.801273 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:35:59.801088    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9336b085-f79b-4e0f-ab70-afe25e1634d0-istio-token\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:36:00.276320 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.276281    2581 generic.go:356] "Generic (PLEG): container finished" podID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerID="7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1" exitCode=0
Sep 16 19:36:00.276498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.276332    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"
Sep 16 19:36:00.276498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.276359    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" event={"ID":"9336b085-f79b-4e0f-ab70-afe25e1634d0","Type":"ContainerDied","Data":"7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1"}
Sep 16 19:36:00.276498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.276415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k" event={"ID":"9336b085-f79b-4e0f-ab70-afe25e1634d0","Type":"ContainerDied","Data":"ede57aba2ef38c849e9d6d280d00e4510057e63f0f6f90f68679afc7c952673b"}
Sep 16 19:36:00.276498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.276439    2581 scope.go:117] "RemoveContainer" containerID="7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1"
Sep 16 19:36:00.285262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.285244    2581 scope.go:117] "RemoveContainer" containerID="85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"
Sep 16 19:36:00.292970 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.292954    2581 scope.go:117] "RemoveContainer" containerID="7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1"
Sep 16 19:36:00.293229 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:36:00.293209    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1\": container with ID starting with 7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1 not found: ID does not exist" containerID="7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1"
Sep 16 19:36:00.293293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.293237    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1"} err="failed to get container status \"7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1\": rpc error: code = NotFound desc = could not find container \"7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1\": container with ID starting with 7fbc48d735584c51f73c8e1d8a9cc48c6241a40575ebf27f96dda95e3f52d7a1 not found: ID does not exist"
Sep 16 19:36:00.293293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.293254    2581 scope.go:117] "RemoveContainer" containerID="85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"
Sep 16 19:36:00.293488 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:36:00.293473    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1\": container with ID starting with 85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1 not found: ID does not exist" containerID="85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"
Sep 16 19:36:00.293533 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.293492    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1"} err="failed to get container status \"85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1\": rpc error: code = NotFound desc = could not find container \"85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1\": container with ID starting with 85bbca2234aca7ca217fdce73ca7e8bfd92d78513da19447ca1e82d7808e70a1 not found: ID does not exist"
Sep 16 19:36:00.302177 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.302154    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"]
Sep 16 19:36:00.310165 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:00.310144    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-5bb8f7bc87-hz88k"]
Sep 16 19:36:01.221444 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:01.221405    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" path="/var/lib/kubelet/pods/9336b085-f79b-4e0f-ab70-afe25e1634d0/volumes"
Sep 16 19:36:03.600624 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600590    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"]
Sep 16 19:36:03.600996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600921    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a633113-ca8c-42f6-8291-e4953f194490" containerName="main"
Sep 16 19:36:03.600996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600933    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a633113-ca8c-42f6-8291-e4953f194490" containerName="main"
Sep 16 19:36:03.600996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy"
Sep 16 19:36:03.600996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy"
Sep 16 19:36:03.600996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600959    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy"
Sep 16 19:36:03.600996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.600965    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy"
Sep 16 19:36:03.601185 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.601010    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy"
Sep 16 19:36:03.601185 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.601020    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a633113-ca8c-42f6-8291-e4953f194490" containerName="main"
Sep 16 19:36:03.601185 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.601108    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9336b085-f79b-4e0f-ab70-afe25e1634d0" containerName="istio-proxy"
Sep 16 19:36:03.605768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.605746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.608580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.608553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:36:03.608736 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.608547    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 16 19:36:03.608736 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.608590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:36:03.608736 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.608614    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-xq94h\""
Sep 16 19:36:03.608736 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.608660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:36:03.608963 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.608914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:36:03.617097 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.617073    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"]
Sep 16 19:36:03.631429 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631553 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631553 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ccbc64-d909-4766-af7c-e05a554ea139-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631553 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631599    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ccbc64-d909-4766-af7c-e05a554ea139-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.631848 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.631824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6w7m\" (UniqueName: \"kubernetes.io/projected/66ccbc64-d909-4766-af7c-e05a554ea139-kube-api-access-f6w7m\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732533 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ccbc64-d909-4766-af7c-e05a554ea139-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ccbc64-d909-4766-af7c-e05a554ea139-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732973 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732973 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.732973 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.732892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6w7m\" (UniqueName: \"kubernetes.io/projected/66ccbc64-d909-4766-af7c-e05a554ea139-kube-api-access-f6w7m\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.733133 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.733043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.733133 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.733086    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.733262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.733141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.733262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.733161    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.733525 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.733509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ccbc64-d909-4766-af7c-e05a554ea139-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.734806 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.734787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.735116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.735097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ccbc64-d909-4766-af7c-e05a554ea139-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.741914 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.741893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6w7m\" (UniqueName: \"kubernetes.io/projected/66ccbc64-d909-4766-af7c-e05a554ea139-kube-api-access-f6w7m\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.883493 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.883408    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"]
Sep 16 19:36:03.888044 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.888024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:03.891135 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.891117    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-nmjwl\""
Sep 16 19:36:03.896843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.896813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"]
Sep 16 19:36:03.916114 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.916089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:03.934525 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.934492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/9a9de67e-6546-4543-8bd6-53a082ff4b20-kube-api-access-djkbk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:03.934627 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:03.934557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9de67e-6546-4543-8bd6-53a082ff4b20-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:04.035600 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.035564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9de67e-6546-4543-8bd6-53a082ff4b20-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:04.035763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.035664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/9a9de67e-6546-4543-8bd6-53a082ff4b20-kube-api-access-djkbk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:04.038107 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.038083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9de67e-6546-4543-8bd6-53a082ff4b20-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:04.045039 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.045015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/9a9de67e-6546-4543-8bd6-53a082ff4b20-kube-api-access-djkbk\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:04.198378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.198345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:04.202346 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.202312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"]
Sep 16 19:36:04.203074 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:36:04.203034    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ccbc64_d909_4766_af7c_e05a554ea139.slice/crio-921cefdb368ac894b2ca163ffd7e78b504367c2b93131d5fdf11b7c81886e702 WatchSource:0}: Error finding container 921cefdb368ac894b2ca163ffd7e78b504367c2b93131d5fdf11b7c81886e702: Status 404 returned error can't find the container with id 921cefdb368ac894b2ca163ffd7e78b504367c2b93131d5fdf11b7c81886e702
Sep 16 19:36:04.301068 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.301029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" event={"ID":"66ccbc64-d909-4766-af7c-e05a554ea139","Type":"ContainerStarted","Data":"df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9"}
Sep 16 19:36:04.301266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.301094    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" event={"ID":"66ccbc64-d909-4766-af7c-e05a554ea139","Type":"ContainerStarted","Data":"921cefdb368ac894b2ca163ffd7e78b504367c2b93131d5fdf11b7c81886e702"}
Sep 16 19:36:04.481212 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:04.481175    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"]
Sep 16 19:36:04.481558 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:36:04.481530    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a9de67e_6546_4543_8bd6_53a082ff4b20.slice/crio-dd3199993a018959e9425327355dd8450b50770618977a53f4614c765517bdba WatchSource:0}: Error finding container dd3199993a018959e9425327355dd8450b50770618977a53f4614c765517bdba: Status 404 returned error can't find the container with id dd3199993a018959e9425327355dd8450b50770618977a53f4614c765517bdba
Sep 16 19:36:05.307930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:05.307881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" event={"ID":"9a9de67e-6546-4543-8bd6-53a082ff4b20","Type":"ContainerStarted","Data":"991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719"}
Sep 16 19:36:05.308415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:05.307948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" event={"ID":"9a9de67e-6546-4543-8bd6-53a082ff4b20","Type":"ContainerStarted","Data":"dd3199993a018959e9425327355dd8450b50770618977a53f4614c765517bdba"}
Sep 16 19:36:05.308415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:05.308132    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:05.327586 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:05.327536    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" podStartSLOduration=2.327522788 podStartE2EDuration="2.327522788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:05.326236466 +0000 UTC m=+3066.630253435" watchObservedRunningTime="2026-09-16 19:36:05.327522788 +0000 UTC m=+3066.631539758"
Sep 16 19:36:07.319116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:07.319082    2581 generic.go:356] "Generic (PLEG): container finished" podID="66ccbc64-d909-4766-af7c-e05a554ea139" containerID="df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9" exitCode=0
Sep 16 19:36:07.319518 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:07.319149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" event={"ID":"66ccbc64-d909-4766-af7c-e05a554ea139","Type":"ContainerDied","Data":"df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9"}
Sep 16 19:36:08.326207 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:08.326173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" event={"ID":"66ccbc64-d909-4766-af7c-e05a554ea139","Type":"ContainerStarted","Data":"915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d"}
Sep 16 19:36:08.350779 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:08.350728    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podStartSLOduration=3.07761309 podStartE2EDuration="5.350713855s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.273100765s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:08.349794928 +0000 UTC m=+3069.653811909" watchObservedRunningTime="2026-09-16 19:36:08.350713855 +0000 UTC m=+3069.654730822"
Sep 16 19:36:08.916793 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:08.916747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:13.917037 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:13.916987    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:36:13.918612 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:13.918575    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:36:23.917321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:23.917274    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:36:33.916647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:33.916594    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:36:36.314519 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:36.314440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:36:43.917547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:43.917500    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:36:53.917284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:36:53.917229    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:37:03.917079 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:37:03.917028    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:37:13.917438 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:37:13.917390    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:37:23.916613 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:37:23.916516    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:37:33.916592 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:37:33.916553    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:37:43.917242 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:37:43.917198    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:37:53.916966 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:37:53.916915    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:38:03.926549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:03.926520    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:38:03.934312 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:03.934286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:38:09.415740 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:09.415702    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"]
Sep 16 19:38:09.433403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:09.433373    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"]
Sep 16 19:38:16.315128 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:16.315089    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:24.439561 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:24.439489    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" containerID="cri-o://915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d" gracePeriod=45
Sep 16 19:38:24.453891 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:24.453835    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main" containerID="cri-o://991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719" gracePeriod=45
Sep 16 19:38:24.927358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:24.927312    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": dial tcp 10.132.0.76:8000: connect: connection refused"
Sep 16 19:38:25.179374 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.179314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:38:25.320544 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-tmp-dir\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.320739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ccbc64-d909-4766-af7c-e05a554ea139-tls-certs\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.320739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-dshm\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.320739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320624    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ccbc64-d909-4766-af7c-e05a554ea139-cabundle-cert\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.320739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6w7m\" (UniqueName: \"kubernetes.io/projected/66ccbc64-d909-4766-af7c-e05a554ea139-kube-api-access-f6w7m\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.320739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-model-cache\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.320739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-home\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.321078 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.320763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-kserve-provision-location\") pod \"66ccbc64-d909-4766-af7c-e05a554ea139\" (UID: \"66ccbc64-d909-4766-af7c-e05a554ea139\") "
Sep 16 19:38:25.321154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.321092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-model-cache" (OuterVolumeSpecName: "model-cache") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:25.321293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.321271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ccbc64-d909-4766-af7c-e05a554ea139-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:38:25.321514 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.321486    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-home" (OuterVolumeSpecName: "home") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:25.323026 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.323002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-dshm" (OuterVolumeSpecName: "dshm") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:25.323206 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.323182    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ccbc64-d909-4766-af7c-e05a554ea139-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:25.323260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.323186    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ccbc64-d909-4766-af7c-e05a554ea139-kube-api-access-f6w7m" (OuterVolumeSpecName: "kube-api-access-f6w7m") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "kube-api-access-f6w7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:25.332907 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.332878    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:25.344046 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.344012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66ccbc64-d909-4766-af7c-e05a554ea139" (UID: "66ccbc64-d909-4766-af7c-e05a554ea139"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:38:25.422262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422232    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422259    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66ccbc64-d909-4766-af7c-e05a554ea139-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422274    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6w7m\" (UniqueName: \"kubernetes.io/projected/66ccbc64-d909-4766-af7c-e05a554ea139-kube-api-access-f6w7m\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422295    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422308    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422319    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422330    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ccbc64-d909-4766-af7c-e05a554ea139-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:25.422437 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:25.422342    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ccbc64-d909-4766-af7c-e05a554ea139-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:26.049322 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.049286    2581 generic.go:356] "Generic (PLEG): container finished" podID="66ccbc64-d909-4766-af7c-e05a554ea139" containerID="915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d" exitCode=0
Sep 16 19:38:26.049710 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.049373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" event={"ID":"66ccbc64-d909-4766-af7c-e05a554ea139","Type":"ContainerDied","Data":"915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d"}
Sep 16 19:38:26.049710 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.049442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx" event={"ID":"66ccbc64-d909-4766-af7c-e05a554ea139","Type":"ContainerDied","Data":"921cefdb368ac894b2ca163ffd7e78b504367c2b93131d5fdf11b7c81886e702"}
Sep 16 19:38:26.049710 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.049375    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"
Sep 16 19:38:26.049710 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.049474    2581 scope.go:117] "RemoveContainer" containerID="915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d"
Sep 16 19:38:26.058892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.058876    2581 scope.go:117] "RemoveContainer" containerID="df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9"
Sep 16 19:38:26.066834 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.066813    2581 scope.go:117] "RemoveContainer" containerID="915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d"
Sep 16 19:38:26.067155 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:38:26.067133    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d\": container with ID starting with 915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d not found: ID does not exist" containerID="915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d"
Sep 16 19:38:26.067233 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.067166    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d"} err="failed to get container status \"915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d\": rpc error: code = NotFound desc = could not find container \"915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d\": container with ID starting with 915c21ccdc42769aa851945dad8fbef6b261f11b585201c30dc4668fa5252d4d not found: ID does not exist"
Sep 16 19:38:26.067233 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.067189    2581 scope.go:117] "RemoveContainer" containerID="df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9"
Sep 16 19:38:26.067465 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:38:26.067447    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9\": container with ID starting with df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9 not found: ID does not exist" containerID="df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9"
Sep 16 19:38:26.067521 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.067472    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9"} err="failed to get container status \"df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9\": rpc error: code = NotFound desc = could not find container \"df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9\": container with ID starting with df003766dbc5b14ee569a2ef1308d7f8c0b100251b93e48b4ff84434540e54b9 not found: ID does not exist"
Sep 16 19:38:26.073962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.073938    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"]
Sep 16 19:38:26.080555 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.080527    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-774db89d85n4cqx"]
Sep 16 19:38:26.314506 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:26.314423    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:27.221548 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:27.221511    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" path="/var/lib/kubelet/pods/66ccbc64-d909-4766-af7c-e05a554ea139/volumes"
Sep 16 19:38:35.413384 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.413345    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"]
Sep 16 19:38:35.413766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.413695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="storage-initializer"
Sep 16 19:38:35.413766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.413708    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="storage-initializer"
Sep 16 19:38:35.413766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.413719    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main"
Sep 16 19:38:35.413766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.413724    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main"
Sep 16 19:38:35.413937 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.413789    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66ccbc64-d909-4766-af7c-e05a554ea139" containerName="main"
Sep 16 19:38:35.418464 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.418441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.421134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.421060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:38:35.421134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.421086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:38:35.421134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.421090    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-dc5sv\""
Sep 16 19:38:35.421426 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.421311    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:38:35.421523 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.421505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:38:35.422221 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.422203    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:38:35.431150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.431129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"]
Sep 16 19:38:35.500922 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.500893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tmp-dir\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.500928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-model-cache\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.500947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-home\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.500994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-dshm\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.501044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kserve-provision-location\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501222 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.501085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tls-certs\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501222 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.501101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95z6t\" (UniqueName: \"kubernetes.io/projected/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kube-api-access-95z6t\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.501222 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.501121    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-cabundle-cert\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602015 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.601984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-dshm\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kserve-provision-location\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tls-certs\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95z6t\" (UniqueName: \"kubernetes.io/projected/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kube-api-access-95z6t\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-cabundle-cert\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602181    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tmp-dir\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-model-cache\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.602547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.602237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-home\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.603052 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.603020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-cabundle-cert\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.603500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.603202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kserve-provision-location\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.603500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.603404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-home\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.603500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.603452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-model-cache\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.603950 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.603920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tmp-dir\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.605406 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.605382    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tls-certs\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.605605 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.605578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-dshm\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.615344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.615321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95z6t\" (UniqueName: \"kubernetes.io/projected/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kube-api-access-95z6t\") pod \"stop-feature-test-kserve-7666f9756b-ffv86\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.621582 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.621559    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"]
Sep 16 19:38:35.626163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.626145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.628851 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.628831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-869cs\""
Sep 16 19:38:35.637469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.637449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"]
Sep 16 19:38:35.703627 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.703594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l56d\" (UniqueName: \"kubernetes.io/projected/7acab2cb-de01-4bf6-8636-0634a39fc4b3-kube-api-access-4l56d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.703775 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.703648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acab2cb-de01-4bf6-8636-0634a39fc4b3-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.730237 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.730211    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:35.804433 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.804397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acab2cb-de01-4bf6-8636-0634a39fc4b3-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.804549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.804481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l56d\" (UniqueName: \"kubernetes.io/projected/7acab2cb-de01-4bf6-8636-0634a39fc4b3-kube-api-access-4l56d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.807368 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.807342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acab2cb-de01-4bf6-8636-0634a39fc4b3-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.813933 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.813913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l56d\" (UniqueName: \"kubernetes.io/projected/7acab2cb-de01-4bf6-8636-0634a39fc4b3-kube-api-access-4l56d\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:35.937199 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:35.937163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:36.014349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:36.014324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"]
Sep 16 19:38:36.015593 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:38:36.015568    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3e5b5f_5eed_477c_be57_b93e965e8c0e.slice/crio-358b38130c1cd586bbed38108e5892b3d9c4d79e2b265ddd97822ed15e333721 WatchSource:0}: Error finding container 358b38130c1cd586bbed38108e5892b3d9c4d79e2b265ddd97822ed15e333721: Status 404 returned error can't find the container with id 358b38130c1cd586bbed38108e5892b3d9c4d79e2b265ddd97822ed15e333721
Sep 16 19:38:36.100544 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:36.100486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" event={"ID":"9a3e5b5f-5eed-477c-be57-b93e965e8c0e","Type":"ContainerStarted","Data":"5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350"}
Sep 16 19:38:36.100714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:36.100569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" event={"ID":"9a3e5b5f-5eed-477c-be57-b93e965e8c0e","Type":"ContainerStarted","Data":"358b38130c1cd586bbed38108e5892b3d9c4d79e2b265ddd97822ed15e333721"}
Sep 16 19:38:36.210010 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:36.209979    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"]
Sep 16 19:38:36.211583 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:38:36.211553    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7acab2cb_de01_4bf6_8636_0634a39fc4b3.slice/crio-b43c80d65cae16a1bee88bca3b6abc93179bfa9b4f692a59d1daa71cd241574f WatchSource:0}: Error finding container b43c80d65cae16a1bee88bca3b6abc93179bfa9b4f692a59d1daa71cd241574f: Status 404 returned error can't find the container with id b43c80d65cae16a1bee88bca3b6abc93179bfa9b4f692a59d1daa71cd241574f
Sep 16 19:38:36.314593 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:36.314559    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:36.314749 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:36.314653    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:38:37.106946 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:37.106903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" event={"ID":"7acab2cb-de01-4bf6-8636-0634a39fc4b3","Type":"ContainerStarted","Data":"6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545"}
Sep 16 19:38:37.107341 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:37.106961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" event={"ID":"7acab2cb-de01-4bf6-8636-0634a39fc4b3","Type":"ContainerStarted","Data":"b43c80d65cae16a1bee88bca3b6abc93179bfa9b4f692a59d1daa71cd241574f"}
Sep 16 19:38:37.107341 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:37.107004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:38:37.123496 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:37.123452    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podStartSLOduration=2.123438864 podStartE2EDuration="2.123438864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:37.12177405 +0000 UTC m=+3218.425791017" watchObservedRunningTime="2026-09-16 19:38:37.123438864 +0000 UTC m=+3218.427455830"
Sep 16 19:38:39.119462 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:39.119429    2581 generic.go:356] "Generic (PLEG): container finished" podID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerID="5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350" exitCode=0
Sep 16 19:38:39.119822 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:39.119487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" event={"ID":"9a3e5b5f-5eed-477c-be57-b93e965e8c0e","Type":"ContainerDied","Data":"5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350"}
Sep 16 19:38:40.126691 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:40.126654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" event={"ID":"9a3e5b5f-5eed-477c-be57-b93e965e8c0e","Type":"ContainerStarted","Data":"712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60"}
Sep 16 19:38:40.152007 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:40.151961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podStartSLOduration=2.871329904 podStartE2EDuration="5.151945357s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.280615453s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:38:40.149539939 +0000 UTC m=+3221.453556908" watchObservedRunningTime="2026-09-16 19:38:40.151945357 +0000 UTC m=+3221.455962324"
Sep 16 19:38:40.730977 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:40.730936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:45.731063 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:45.731027    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:38:45.732709 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:45.732681    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:38:46.315331 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:46.315281    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:54.705443 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.705416    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:38:54.767188 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.767102    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/9a9de67e-6546-4543-8bd6-53a082ff4b20-kube-api-access-djkbk\") pod \"9a9de67e-6546-4543-8bd6-53a082ff4b20\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") "
Sep 16 19:38:54.767344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.767195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9de67e-6546-4543-8bd6-53a082ff4b20-tls-certs\") pod \"9a9de67e-6546-4543-8bd6-53a082ff4b20\" (UID: \"9a9de67e-6546-4543-8bd6-53a082ff4b20\") "
Sep 16 19:38:54.769204 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.769165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9de67e-6546-4543-8bd6-53a082ff4b20-kube-api-access-djkbk" (OuterVolumeSpecName: "kube-api-access-djkbk") pod "9a9de67e-6546-4543-8bd6-53a082ff4b20" (UID: "9a9de67e-6546-4543-8bd6-53a082ff4b20"). InnerVolumeSpecName "kube-api-access-djkbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:54.769312 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.769217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9de67e-6546-4543-8bd6-53a082ff4b20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a9de67e-6546-4543-8bd6-53a082ff4b20" (UID: "9a9de67e-6546-4543-8bd6-53a082ff4b20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:54.868090 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.868057    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkbk\" (UniqueName: \"kubernetes.io/projected/9a9de67e-6546-4543-8bd6-53a082ff4b20-kube-api-access-djkbk\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:54.868090 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:54.868087    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9de67e-6546-4543-8bd6-53a082ff4b20-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:38:55.202386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.202348    2581 generic.go:356] "Generic (PLEG): container finished" podID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerID="991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719" exitCode=0
Sep 16 19:38:55.202581 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.202393    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"
Sep 16 19:38:55.202581 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.202428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" event={"ID":"9a9de67e-6546-4543-8bd6-53a082ff4b20","Type":"ContainerDied","Data":"991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719"}
Sep 16 19:38:55.202581 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.202477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj" event={"ID":"9a9de67e-6546-4543-8bd6-53a082ff4b20","Type":"ContainerDied","Data":"dd3199993a018959e9425327355dd8450b50770618977a53f4614c765517bdba"}
Sep 16 19:38:55.202581 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.202493    2581 scope.go:117] "RemoveContainer" containerID="991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719"
Sep 16 19:38:55.212436 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.212421    2581 scope.go:117] "RemoveContainer" containerID="991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719"
Sep 16 19:38:55.212726 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:38:55.212707    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719\": container with ID starting with 991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719 not found: ID does not exist" containerID="991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719"
Sep 16 19:38:55.212767 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.212735    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719"} err="failed to get container status \"991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719\": rpc error: code = NotFound desc = could not find container \"991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719\": container with ID starting with 991deb051511b4b93fe8ed04d78390d83c671d6ab8123c095480d3f217a91719 not found: ID does not exist"
Sep 16 19:38:55.226128 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.226105    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"]
Sep 16 19:38:55.231551 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.231527    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexggkj"]
Sep 16 19:38:55.730656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:55.730611    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:38:57.226530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:38:57.226493    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" path="/var/lib/kubelet/pods/9a9de67e-6546-4543-8bd6-53a082ff4b20/volumes"
Sep 16 19:39:05.731232 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.731187    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:39:05.986996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.986883    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"]
Sep 16 19:39:05.987986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.987958    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main"
Sep 16 19:39:05.987986 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.987985    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main"
Sep 16 19:39:05.988175 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.988148    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9de67e-6546-4543-8bd6-53a082ff4b20" containerName="main"
Sep 16 19:39:05.993171 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.993148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:05.996497 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.996427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-p9thc\""
Sep 16 19:39:05.996614 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.996502    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:39:05.997378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.997358    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:39:05.997493 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.997412    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-49hcg\""
Sep 16 19:39:05.997794 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.997765    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:39:05.999808 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:05.999788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"]
Sep 16 19:39:06.063080 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.063047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e0d85-1235-49fd-8249-9a7f6324bd48-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.063264 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.063116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbjvr\" (UniqueName: \"kubernetes.io/projected/b45e0d85-1235-49fd-8249-9a7f6324bd48-kube-api-access-gbjvr\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.163810 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.163777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbjvr\" (UniqueName: \"kubernetes.io/projected/b45e0d85-1235-49fd-8249-9a7f6324bd48-kube-api-access-gbjvr\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.164034 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.163874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e0d85-1235-49fd-8249-9a7f6324bd48-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.166579 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.166557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e0d85-1235-49fd-8249-9a7f6324bd48-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.172322 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.172298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbjvr\" (UniqueName: \"kubernetes.io/projected/b45e0d85-1235-49fd-8249-9a7f6324bd48-kube-api-access-gbjvr\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.305741 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.305649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:06.581662 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:06.581630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"]
Sep 16 19:39:07.274158 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:07.274122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" event={"ID":"b45e0d85-1235-49fd-8249-9a7f6324bd48","Type":"ContainerStarted","Data":"7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793"}
Sep 16 19:39:07.274545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:07.274173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" event={"ID":"b45e0d85-1235-49fd-8249-9a7f6324bd48","Type":"ContainerStarted","Data":"81528b86a1eb96cea3e832cd55f61799424c024fbed59898a2671dc7e909dde5"}
Sep 16 19:39:07.274545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:07.274216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:07.291924 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:07.291880    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" podStartSLOduration=2.291850986 podStartE2EDuration="2.291850986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:07.29044377 +0000 UTC m=+3248.594460741" watchObservedRunningTime="2026-09-16 19:39:07.291850986 +0000 UTC m=+3248.595867952"
Sep 16 19:39:08.115602 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:08.115567    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:39:15.731527 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:15.731477    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:39:25.731468 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:25.731415    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:39:35.731472 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:35.731428    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:39:38.280896 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:38.280842    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:39:45.731660 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:45.731614    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:39:55.730879 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:55.730802    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:39:59.475015 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:59.474987    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:39:59.478356 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:59.478333    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:39:59.486469 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:59.486445    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:39:59.489763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:39:59.489743    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:40:05.731530 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:05.731485    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:40:15.731500 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:15.731451    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:40:25.731529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:25.731433    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": dial tcp 10.132.0.78:8000: connect: connection refused"
Sep 16 19:40:35.741003 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:35.740970    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:40:35.748987 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:35.748959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:40:37.486670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:37.486631    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"]
Sep 16 19:40:37.502828 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:37.502798    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"]
Sep 16 19:40:38.115454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:38.115412    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:48.114838 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:48.114797    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:52.521760 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:52.521712    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main" containerID="cri-o://6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545" gracePeriod=45
Sep 16 19:40:52.773638 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:52.773551    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main" containerID="cri-o://712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60" gracePeriod=45
Sep 16 19:40:53.528567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.528542    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:40:53.596749 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-cabundle-cert\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.596749 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596715    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-model-cache\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.596749 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596745    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kserve-provision-location\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.597009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tls-certs\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.597009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596790    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tmp-dir\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.597009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596805    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-dshm\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.597009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95z6t\" (UniqueName: \"kubernetes.io/projected/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kube-api-access-95z6t\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.597009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.596943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-home\") pod \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\" (UID: \"9a3e5b5f-5eed-477c-be57-b93e965e8c0e\") "
Sep 16 19:40:53.597256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.597028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:53.597256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.597075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:53.597256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.597208    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.597256 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.597229    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.597645 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.597621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-home" (OuterVolumeSpecName: "home") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:53.599200 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.599168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:40:53.599200 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.599168    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kube-api-access-95z6t" (OuterVolumeSpecName: "kube-api-access-95z6t") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "kube-api-access-95z6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:53.599349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.599295    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-dshm" (OuterVolumeSpecName: "dshm") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:53.608734 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.608711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:53.625620 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.625596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a3e5b5f-5eed-477c-be57-b93e965e8c0e" (UID: "9a3e5b5f-5eed-477c-be57-b93e965e8c0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:53.697923 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.697886    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.697923 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.697920    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.697923 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.697931    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.698100 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.697939    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.698100 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.697947    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95z6t\" (UniqueName: \"kubernetes.io/projected/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-kube-api-access-95z6t\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.698100 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.697955    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a3e5b5f-5eed-477c-be57-b93e965e8c0e-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:40:53.835843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.835806    2581 generic.go:356] "Generic (PLEG): container finished" podID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerID="712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60" exitCode=0
Sep 16 19:40:53.836034 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.835899    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"
Sep 16 19:40:53.836034 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.835898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" event={"ID":"9a3e5b5f-5eed-477c-be57-b93e965e8c0e","Type":"ContainerDied","Data":"712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60"}
Sep 16 19:40:53.836034 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.835965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86" event={"ID":"9a3e5b5f-5eed-477c-be57-b93e965e8c0e","Type":"ContainerDied","Data":"358b38130c1cd586bbed38108e5892b3d9c4d79e2b265ddd97822ed15e333721"}
Sep 16 19:40:53.836034 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.835997    2581 scope.go:117] "RemoveContainer" containerID="712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60"
Sep 16 19:40:53.846258 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.846234    2581 scope.go:117] "RemoveContainer" containerID="5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350"
Sep 16 19:40:53.854183 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.854163    2581 scope.go:117] "RemoveContainer" containerID="712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60"
Sep 16 19:40:53.854461 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:40:53.854439    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60\": container with ID starting with 712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60 not found: ID does not exist" containerID="712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60"
Sep 16 19:40:53.854541 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.854467    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60"} err="failed to get container status \"712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60\": rpc error: code = NotFound desc = could not find container \"712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60\": container with ID starting with 712f8b4940115f35d51cfcd959f51d7245b1b1ab090fef9096e3c9226330af60 not found: ID does not exist"
Sep 16 19:40:53.854541 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.854482    2581 scope.go:117] "RemoveContainer" containerID="5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350"
Sep 16 19:40:53.854717 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:40:53.854697    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350\": container with ID starting with 5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350 not found: ID does not exist" containerID="5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350"
Sep 16 19:40:53.854757 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.854724    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350"} err="failed to get container status \"5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350\": rpc error: code = NotFound desc = could not find container \"5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350\": container with ID starting with 5fbef648c7b7e4cc857ed29c4c2c261d5efc8f096e4336c6b0e414ffbfa4e350 not found: ID does not exist"
Sep 16 19:40:53.861965 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.861942    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"]
Sep 16 19:40:53.866636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:53.866613    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-ffv86"]
Sep 16 19:40:55.221446 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:55.221411    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" path="/var/lib/kubelet/pods/9a3e5b5f-5eed-477c-be57-b93e965e8c0e/volumes"
Sep 16 19:40:58.114641 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:58.114599    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:40:58.115046 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:40:58.114713    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:41:03.837297 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.837264    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"]
Sep 16 19:41:03.837654 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.837581    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="storage-initializer"
Sep 16 19:41:03.837654 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.837592    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="storage-initializer"
Sep 16 19:41:03.837654 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.837605    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main"
Sep 16 19:41:03.837654 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.837610    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main"
Sep 16 19:41:03.837792 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.837668    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3e5b5f-5eed-477c-be57-b93e965e8c0e" containerName="main"
Sep 16 19:41:03.842473 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.842451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.845232 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.845208    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 19:41:03.845318 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.845281    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:41:03.851894 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.851874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"]
Sep 16 19:41:03.873721 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-model-cache\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.873882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmj4h\" (UniqueName: \"kubernetes.io/projected/9d80690d-7814-4faf-a49b-7c48711383a1-kube-api-access-rmj4h\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.873882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-kserve-provision-location\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.873882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-dshm\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.873882 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d80690d-7814-4faf-a49b-7c48711383a1-cabundle-cert\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.874098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d80690d-7814-4faf-a49b-7c48711383a1-tls-certs\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.874098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-tmp-dir\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.874098 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.873988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-home\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-dshm\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974636 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d80690d-7814-4faf-a49b-7c48711383a1-cabundle-cert\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d80690d-7814-4faf-a49b-7c48711383a1-tls-certs\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-tmp-dir\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974732    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-home\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-model-cache\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmj4h\" (UniqueName: \"kubernetes.io/projected/9d80690d-7814-4faf-a49b-7c48711383a1-kube-api-access-rmj4h\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.974843 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.974839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-kserve-provision-location\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.975196 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.975155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-tmp-dir\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.975326 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.975308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-model-cache\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.975386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.975328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-home\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.975491 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.975473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d80690d-7814-4faf-a49b-7c48711383a1-cabundle-cert\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.975539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.975523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-kserve-provision-location\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.977447 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.977423    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-dshm\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.977840 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.977817    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d80690d-7814-4faf-a49b-7c48711383a1-tls-certs\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:03.984481 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:03.984458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmj4h\" (UniqueName: \"kubernetes.io/projected/9d80690d-7814-4faf-a49b-7c48711383a1-kube-api-access-rmj4h\") pod \"stop-feature-test-kserve-7666f9756b-lm5lf\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:04.153575 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:04.153543    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:04.435366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:04.435342    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"]
Sep 16 19:41:04.437370 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:41:04.437344    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d80690d_7814_4faf_a49b_7c48711383a1.slice/crio-968c606ce44c3eb4d85039b187c471a8712ebc30781c3b84258edfc5ed9c3506 WatchSource:0}: Error finding container 968c606ce44c3eb4d85039b187c471a8712ebc30781c3b84258edfc5ed9c3506: Status 404 returned error can't find the container with id 968c606ce44c3eb4d85039b187c471a8712ebc30781c3b84258edfc5ed9c3506
Sep 16 19:41:04.897293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:04.897194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" event={"ID":"9d80690d-7814-4faf-a49b-7c48711383a1","Type":"ContainerStarted","Data":"a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5"}
Sep 16 19:41:04.897712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:04.897292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" event={"ID":"9d80690d-7814-4faf-a49b-7c48711383a1","Type":"ContainerStarted","Data":"968c606ce44c3eb4d85039b187c471a8712ebc30781c3b84258edfc5ed9c3506"}
Sep 16 19:41:06.909068 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:06.909024    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d80690d-7814-4faf-a49b-7c48711383a1" containerID="a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5" exitCode=0
Sep 16 19:41:06.909068 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:06.909053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" event={"ID":"9d80690d-7814-4faf-a49b-7c48711383a1","Type":"ContainerDied","Data":"a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5"}
Sep 16 19:41:07.915958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:07.915916    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" event={"ID":"9d80690d-7814-4faf-a49b-7c48711383a1","Type":"ContainerStarted","Data":"91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63"}
Sep 16 19:41:07.945316 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:07.945259    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podStartSLOduration=2.707001936 podStartE2EDuration="4.945243805s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.238241869s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:07.941608514 +0000 UTC m=+3369.245625482" watchObservedRunningTime="2026-09-16 19:41:07.945243805 +0000 UTC m=+3369.249260771"
Sep 16 19:41:08.114764 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:08.114727    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:08.154448 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:08.154413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:14.154250 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:14.154207    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:41:14.155980 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:14.155950    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:41:18.115649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:18.115606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:22.770301 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.770274    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:41:22.834502 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.834422    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acab2cb-de01-4bf6-8636-0634a39fc4b3-tls-certs\") pod \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") "
Sep 16 19:41:22.834669 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.834514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4l56d\" (UniqueName: \"kubernetes.io/projected/7acab2cb-de01-4bf6-8636-0634a39fc4b3-kube-api-access-4l56d\") pod \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\" (UID: \"7acab2cb-de01-4bf6-8636-0634a39fc4b3\") "
Sep 16 19:41:22.836639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.836601    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7acab2cb-de01-4bf6-8636-0634a39fc4b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7acab2cb-de01-4bf6-8636-0634a39fc4b3" (UID: "7acab2cb-de01-4bf6-8636-0634a39fc4b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:22.836767 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.836671    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7acab2cb-de01-4bf6-8636-0634a39fc4b3-kube-api-access-4l56d" (OuterVolumeSpecName: "kube-api-access-4l56d") pod "7acab2cb-de01-4bf6-8636-0634a39fc4b3" (UID: "7acab2cb-de01-4bf6-8636-0634a39fc4b3"). InnerVolumeSpecName "kube-api-access-4l56d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:22.935686 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.935652    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l56d\" (UniqueName: \"kubernetes.io/projected/7acab2cb-de01-4bf6-8636-0634a39fc4b3-kube-api-access-4l56d\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:22.935686 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.935685    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7acab2cb-de01-4bf6-8636-0634a39fc4b3-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:41:22.992126 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.992097    2581 generic.go:356] "Generic (PLEG): container finished" podID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerID="6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545" exitCode=0
Sep 16 19:41:22.992258 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.992144    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"
Sep 16 19:41:22.992258 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.992148    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" event={"ID":"7acab2cb-de01-4bf6-8636-0634a39fc4b3","Type":"ContainerDied","Data":"6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545"}
Sep 16 19:41:22.992337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.992270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw" event={"ID":"7acab2cb-de01-4bf6-8636-0634a39fc4b3","Type":"ContainerDied","Data":"b43c80d65cae16a1bee88bca3b6abc93179bfa9b4f692a59d1daa71cd241574f"}
Sep 16 19:41:22.992337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:22.992305    2581 scope.go:117] "RemoveContainer" containerID="6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545"
Sep 16 19:41:23.001182 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:23.001158    2581 scope.go:117] "RemoveContainer" containerID="6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545"
Sep 16 19:41:23.001451 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:41:23.001433    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545\": container with ID starting with 6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545 not found: ID does not exist" containerID="6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545"
Sep 16 19:41:23.001497 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:23.001459    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545"} err="failed to get container status \"6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545\": rpc error: code = NotFound desc = could not find container \"6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545\": container with ID starting with 6c0d713edf83e097ad465f8d08d1c3cbffa2e58fe2df5bb8e1ef582454dd0545 not found: ID does not exist"
Sep 16 19:41:23.014262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:23.014232    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"]
Sep 16 19:41:23.018145 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:23.018121    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-s4mpw"]
Sep 16 19:41:23.223129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:23.223095    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" path="/var/lib/kubelet/pods/7acab2cb-de01-4bf6-8636-0634a39fc4b3/volumes"
Sep 16 19:41:24.154156 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:24.154109    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:41:31.449244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:31.449204    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"]
Sep 16 19:41:34.154798 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:34.154754    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:41:38.280944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:38.280905    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:44.154587 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:44.154545    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:41:46.468402 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:46.468335    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main" containerID="cri-o://7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793" gracePeriod=45
Sep 16 19:41:48.281209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:48.281164    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:54.154279 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:54.154190    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:41:58.281041 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:58.280993    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:58.281445 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:41:58.281116    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:42:04.154434 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:04.154387    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:42:08.281549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:08.281503    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:42:14.154245 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:14.154203    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:42:16.714157 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.714125    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:42:16.789550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.789469    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e0d85-1235-49fd-8249-9a7f6324bd48-tls-certs\") pod \"b45e0d85-1235-49fd-8249-9a7f6324bd48\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") "
Sep 16 19:42:16.789685 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.789547    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbjvr\" (UniqueName: \"kubernetes.io/projected/b45e0d85-1235-49fd-8249-9a7f6324bd48-kube-api-access-gbjvr\") pod \"b45e0d85-1235-49fd-8249-9a7f6324bd48\" (UID: \"b45e0d85-1235-49fd-8249-9a7f6324bd48\") "
Sep 16 19:42:16.791630 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.791600    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b45e0d85-1235-49fd-8249-9a7f6324bd48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b45e0d85-1235-49fd-8249-9a7f6324bd48" (UID: "b45e0d85-1235-49fd-8249-9a7f6324bd48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:16.791764 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.791651    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45e0d85-1235-49fd-8249-9a7f6324bd48-kube-api-access-gbjvr" (OuterVolumeSpecName: "kube-api-access-gbjvr") pod "b45e0d85-1235-49fd-8249-9a7f6324bd48" (UID: "b45e0d85-1235-49fd-8249-9a7f6324bd48"). InnerVolumeSpecName "kube-api-access-gbjvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:16.890697 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.890660    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbjvr\" (UniqueName: \"kubernetes.io/projected/b45e0d85-1235-49fd-8249-9a7f6324bd48-kube-api-access-gbjvr\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:42:16.890697 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:16.890687    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b45e0d85-1235-49fd-8249-9a7f6324bd48-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:42:17.274131 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.274095    2581 generic.go:356] "Generic (PLEG): container finished" podID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerID="7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793" exitCode=0
Sep 16 19:42:17.274338 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.274143    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"
Sep 16 19:42:17.274338 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.274176    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" event={"ID":"b45e0d85-1235-49fd-8249-9a7f6324bd48","Type":"ContainerDied","Data":"7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793"}
Sep 16 19:42:17.274338 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.274233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx" event={"ID":"b45e0d85-1235-49fd-8249-9a7f6324bd48","Type":"ContainerDied","Data":"81528b86a1eb96cea3e832cd55f61799424c024fbed59898a2671dc7e909dde5"}
Sep 16 19:42:17.274338 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.274254    2581 scope.go:117] "RemoveContainer" containerID="7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793"
Sep 16 19:42:17.283343 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.283325    2581 scope.go:117] "RemoveContainer" containerID="7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793"
Sep 16 19:42:17.283614 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:42:17.283595    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793\": container with ID starting with 7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793 not found: ID does not exist" containerID="7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793"
Sep 16 19:42:17.283696 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.283617    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793"} err="failed to get container status \"7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793\": rpc error: code = NotFound desc = could not find container \"7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793\": container with ID starting with 7626a6d55a653eda69e05f70386d25cbc204d3f6639cb6bc877f39389f32c793 not found: ID does not exist"
Sep 16 19:42:17.293205 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.293182    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"]
Sep 16 19:42:17.296489 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:17.296467    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheghxzx"]
Sep 16 19:42:19.223516 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:19.223481    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" path="/var/lib/kubelet/pods/b45e0d85-1235-49fd-8249-9a7f6324bd48/volumes"
Sep 16 19:42:24.154345 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:24.154298    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:42:34.154498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:34.154449    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:42:41.035777 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.035740    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"]
Sep 16 19:42:41.036306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.036247    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main"
Sep 16 19:42:41.036306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.036267    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main"
Sep 16 19:42:41.036306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.036289    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main"
Sep 16 19:42:41.036306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.036297    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main"
Sep 16 19:42:41.036522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.036380    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b45e0d85-1235-49fd-8249-9a7f6324bd48" containerName="main"
Sep 16 19:42:41.036522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.036395    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7acab2cb-de01-4bf6-8636-0634a39fc4b3" containerName="main"
Sep 16 19:42:41.039623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.039598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.043558 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.043532    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 16 19:42:41.043703 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.043683    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 16 19:42:41.043891 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.043848    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-sjwhv\""
Sep 16 19:42:41.044042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.043915    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 16 19:42:41.044158 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.043848    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-tchjm\""
Sep 16 19:42:41.049071 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.049051    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"]
Sep 16 19:42:41.095052 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.095021    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52af8ec5-4f53-48cd-91f2-a8686f10331c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.095218 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.095088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4gc\" (UniqueName: \"kubernetes.io/projected/52af8ec5-4f53-48cd-91f2-a8686f10331c-kube-api-access-7x4gc\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.195803 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.195764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4gc\" (UniqueName: \"kubernetes.io/projected/52af8ec5-4f53-48cd-91f2-a8686f10331c-kube-api-access-7x4gc\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.196039 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.195817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52af8ec5-4f53-48cd-91f2-a8686f10331c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.198647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.198615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52af8ec5-4f53-48cd-91f2-a8686f10331c-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.204386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.204362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4gc\" (UniqueName: \"kubernetes.io/projected/52af8ec5-4f53-48cd-91f2-a8686f10331c-kube-api-access-7x4gc\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.351708 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.351618    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:41.631148 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:41.631114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"]
Sep 16 19:42:41.632674 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:42:41.632645    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52af8ec5_4f53_48cd_91f2_a8686f10331c.slice/crio-34dcaf3282d44d3dff8ba2be0d9ba04e136b0ba46e125b910b9af7f56d46c0fc WatchSource:0}: Error finding container 34dcaf3282d44d3dff8ba2be0d9ba04e136b0ba46e125b910b9af7f56d46c0fc: Status 404 returned error can't find the container with id 34dcaf3282d44d3dff8ba2be0d9ba04e136b0ba46e125b910b9af7f56d46c0fc
Sep 16 19:42:42.418494 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:42.418458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" event={"ID":"52af8ec5-4f53-48cd-91f2-a8686f10331c","Type":"ContainerStarted","Data":"ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec"}
Sep 16 19:42:42.418903 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:42.418519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:42:42.418903 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:42.418538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" event={"ID":"52af8ec5-4f53-48cd-91f2-a8686f10331c","Type":"ContainerStarted","Data":"34dcaf3282d44d3dff8ba2be0d9ba04e136b0ba46e125b910b9af7f56d46c0fc"}
Sep 16 19:42:42.435806 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:42.435750    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" podStartSLOduration=1.435730064 podStartE2EDuration="1.435730064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:42:42.434730504 +0000 UTC m=+3463.738747476" watchObservedRunningTime="2026-09-16 19:42:42.435730064 +0000 UTC m=+3463.739747033"
Sep 16 19:42:44.154168 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:44.154128    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:42:54.154764 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:42:54.154716    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:43:04.163598 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:04.163562    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:43:04.171242 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:04.171219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:43:05.790180 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:05.790140    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"]
Sep 16 19:43:13.425138 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:13.425106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:43:20.819493 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:20.819431    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" containerID="cri-o://91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63" gracePeriod=45
Sep 16 19:43:21.164467 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.164423    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Sep 16 19:43:21.469681 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.469656    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:43:21.617071 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.616989    2581 generic.go:356] "Generic (PLEG): container finished" podID="9d80690d-7814-4faf-a49b-7c48711383a1" containerID="91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63" exitCode=0
Sep 16 19:43:21.617071 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.617039    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"
Sep 16 19:43:21.617280 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.617089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" event={"ID":"9d80690d-7814-4faf-a49b-7c48711383a1","Type":"ContainerDied","Data":"91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63"}
Sep 16 19:43:21.617280 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.617141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf" event={"ID":"9d80690d-7814-4faf-a49b-7c48711383a1","Type":"ContainerDied","Data":"968c606ce44c3eb4d85039b187c471a8712ebc30781c3b84258edfc5ed9c3506"}
Sep 16 19:43:21.617280 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.617178    2581 scope.go:117] "RemoveContainer" containerID="91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63"
Sep 16 19:43:21.625508 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.625488    2581 scope.go:117] "RemoveContainer" containerID="a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5"
Sep 16 19:43:21.632875 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.632837    2581 scope.go:117] "RemoveContainer" containerID="91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63"
Sep 16 19:43:21.633029 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-dshm\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633044    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-kserve-provision-location\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d80690d-7814-4faf-a49b-7c48711383a1-tls-certs\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-home\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633139 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:43:21.633125    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63\": container with ID starting with 91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63 not found: ID does not exist" containerID="91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63"
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmj4h\" (UniqueName: \"kubernetes.io/projected/9d80690d-7814-4faf-a49b-7c48711383a1-kube-api-access-rmj4h\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633155    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63"} err="failed to get container status \"91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63\": rpc error: code = NotFound desc = could not find container \"91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63\": container with ID starting with 91d3d1034509b9c77ead3eb45e3b5fbbaa6ea72845b3adc8a00c6d28bf4feb63 not found: ID does not exist"
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633178    2581 scope.go:117] "RemoveContainer" containerID="a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5"
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-tmp-dir\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633234    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d80690d-7814-4faf-a49b-7c48711383a1-cabundle-cert\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633265    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-model-cache\") pod \"9d80690d-7814-4faf-a49b-7c48711383a1\" (UID: \"9d80690d-7814-4faf-a49b-7c48711383a1\") "
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:43:21.633519    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5\": container with ID starting with a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5 not found: ID does not exist" containerID="a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5"
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633559    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5"} err="failed to get container status \"a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5\": rpc error: code = NotFound desc = could not find container \"a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5\": container with ID starting with a8714c0c1e105afe5c8985ca9c73fee6a854c59fb89f8e2091504bf7956246f5 not found: ID does not exist"
Sep 16 19:43:21.633768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633686    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:21.634282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-home" (OuterVolumeSpecName: "home") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:21.634282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.633957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d80690d-7814-4faf-a49b-7c48711383a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:43:21.635132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.635103    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-dshm" (OuterVolumeSpecName: "dshm") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:21.635285 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.635266    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d80690d-7814-4faf-a49b-7c48711383a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:21.635694 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.635674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d80690d-7814-4faf-a49b-7c48711383a1-kube-api-access-rmj4h" (OuterVolumeSpecName: "kube-api-access-rmj4h") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "kube-api-access-rmj4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:21.645766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.645734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:21.657825 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.657802    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9d80690d-7814-4faf-a49b-7c48711383a1" (UID: "9d80690d-7814-4faf-a49b-7c48711383a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:43:21.734639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734574    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmj4h\" (UniqueName: \"kubernetes.io/projected/9d80690d-7814-4faf-a49b-7c48711383a1-kube-api-access-rmj4h\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734598    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734611    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9d80690d-7814-4faf-a49b-7c48711383a1-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734625    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734633    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734639 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734642    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734927 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734650    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d80690d-7814-4faf-a49b-7c48711383a1-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.734927 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.734658    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d80690d-7814-4faf-a49b-7c48711383a1-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:43:21.941849 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.941821    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"]
Sep 16 19:43:21.946065 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:21.946039    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-7666f9756b-lm5lf"]
Sep 16 19:43:23.221169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:43:23.221133    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" path="/var/lib/kubelet/pods/9d80690d-7814-4faf-a49b-7c48711383a1/volumes"
Sep 16 19:44:06.897825 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.897786    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"]
Sep 16 19:44:06.898393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.898259    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main"
Sep 16 19:44:06.898393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.898280    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main"
Sep 16 19:44:06.898393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.898300    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="storage-initializer"
Sep 16 19:44:06.898393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.898308    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="storage-initializer"
Sep 16 19:44:06.898535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.898399    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d80690d-7814-4faf-a49b-7c48711383a1" containerName="main"
Sep 16 19:44:06.902925 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.902904    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:06.905957 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.905928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 16 19:44:06.906076 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.906024    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:44:06.906076 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.906051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-52sr6\""
Sep 16 19:44:06.906336 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.906317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 16 19:44:06.906382 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.906367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:44:06.907086 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.907071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 16 19:44:06.914424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:06.914398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"]
Sep 16 19:44:07.006225 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-cabundle-cert\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tmp-dir\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-dshm\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h2l6\" (UniqueName: \"kubernetes.io/projected/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kube-api-access-4h2l6\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006385 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tls-certs\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-home\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.006534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.006451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-model-cache\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107740 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-cabundle-cert\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tmp-dir\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-dshm\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h2l6\" (UniqueName: \"kubernetes.io/projected/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kube-api-access-4h2l6\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tls-certs\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-home\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.107976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.107938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-model-cache\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.108384 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.108254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tmp-dir\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.108384 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.108307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.108498 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.108448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-model-cache\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.108559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.108489    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-home\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.108899 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.108852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-cabundle-cert\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.110149 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.110125    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-dshm\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.110410 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.110392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tls-certs\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.116117 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.116087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h2l6\" (UniqueName: \"kubernetes.io/projected/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kube-api-access-4h2l6\") pod \"lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.214003 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.213967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:07.494231 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.494199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"]
Sep 16 19:44:07.495616 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:44:07.495589    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50e09fc_b2d4_48b8_8c36_f7e53c3d9f7f.slice/crio-644de9db0ece531723a9702db6a9073ae70cb0bfa1453a827d472c3943c3ce0f WatchSource:0}: Error finding container 644de9db0ece531723a9702db6a9073ae70cb0bfa1453a827d472c3943c3ce0f: Status 404 returned error can't find the container with id 644de9db0ece531723a9702db6a9073ae70cb0bfa1453a827d472c3943c3ce0f
Sep 16 19:44:07.839335 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.839234    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" event={"ID":"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f","Type":"ContainerStarted","Data":"82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd"}
Sep 16 19:44:07.839335 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:07.839287    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" event={"ID":"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f","Type":"ContainerStarted","Data":"644de9db0ece531723a9702db6a9073ae70cb0bfa1453a827d472c3943c3ce0f"}
Sep 16 19:44:10.858826 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:10.858791    2581 generic.go:356] "Generic (PLEG): container finished" podID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerID="82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd" exitCode=0
Sep 16 19:44:10.859241 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:10.858893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" event={"ID":"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f","Type":"ContainerDied","Data":"82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd"}
Sep 16 19:44:11.866690 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:11.866651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" event={"ID":"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f","Type":"ContainerStarted","Data":"d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5"}
Sep 16 19:44:11.891964 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:11.891911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podStartSLOduration=3.167714041 podStartE2EDuration="5.891894268s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.724180227s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:44:11.889037989 +0000 UTC m=+3553.193054957" watchObservedRunningTime="2026-09-16 19:44:11.891894268 +0000 UTC m=+3553.195911235"
Sep 16 19:44:12.214460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:12.214428    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:17.214942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:17.214893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:44:17.216547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:17.216488    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:44:27.215329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:27.215285    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:44:37.215111 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:37.215059    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:44:47.215035 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:47.214988    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:44:56.750768 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:56.750690    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"]
Sep 16 19:44:57.214852 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:57.214810    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:44:59.513722 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:59.513686    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:44:59.517209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:59.517188    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:44:59.524174 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:59.524155    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:44:59.527308 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:44:59.527289    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:45:03.425937 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:03.425887    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:07.215307 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:07.215254    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:45:11.771208 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:11.771161    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main" containerID="cri-o://ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec" gracePeriod=45
Sep 16 19:45:13.425356 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:13.425317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:17.215343 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:17.215303    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Sep 16 19:45:23.425387 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:23.425348    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:23.425892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:23.425456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:45:27.225626 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:27.225589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:45:27.233991 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:27.233963    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:45:33.425343 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:33.425293    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:45:42.022099 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.022074    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:45:42.088620 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.088543    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x4gc\" (UniqueName: \"kubernetes.io/projected/52af8ec5-4f53-48cd-91f2-a8686f10331c-kube-api-access-7x4gc\") pod \"52af8ec5-4f53-48cd-91f2-a8686f10331c\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") "
Sep 16 19:45:42.088620 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.088608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52af8ec5-4f53-48cd-91f2-a8686f10331c-tls-certs\") pod \"52af8ec5-4f53-48cd-91f2-a8686f10331c\" (UID: \"52af8ec5-4f53-48cd-91f2-a8686f10331c\") "
Sep 16 19:45:42.090789 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.090756    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52af8ec5-4f53-48cd-91f2-a8686f10331c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52af8ec5-4f53-48cd-91f2-a8686f10331c" (UID: "52af8ec5-4f53-48cd-91f2-a8686f10331c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:45:42.090932 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.090884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52af8ec5-4f53-48cd-91f2-a8686f10331c-kube-api-access-7x4gc" (OuterVolumeSpecName: "kube-api-access-7x4gc") pod "52af8ec5-4f53-48cd-91f2-a8686f10331c" (UID: "52af8ec5-4f53-48cd-91f2-a8686f10331c"). InnerVolumeSpecName "kube-api-access-7x4gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:45:42.189758 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.189718    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x4gc\" (UniqueName: \"kubernetes.io/projected/52af8ec5-4f53-48cd-91f2-a8686f10331c-kube-api-access-7x4gc\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:42.189908 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.189781    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52af8ec5-4f53-48cd-91f2-a8686f10331c-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:45:42.346621 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.346531    2581 generic.go:356] "Generic (PLEG): container finished" podID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerID="ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec" exitCode=0
Sep 16 19:45:42.346621 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.346588    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"
Sep 16 19:45:42.346801 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.346590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" event={"ID":"52af8ec5-4f53-48cd-91f2-a8686f10331c","Type":"ContainerDied","Data":"ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec"}
Sep 16 19:45:42.346801 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.346722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw" event={"ID":"52af8ec5-4f53-48cd-91f2-a8686f10331c","Type":"ContainerDied","Data":"34dcaf3282d44d3dff8ba2be0d9ba04e136b0ba46e125b910b9af7f56d46c0fc"}
Sep 16 19:45:42.346801 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.346747    2581 scope.go:117] "RemoveContainer" containerID="ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec"
Sep 16 19:45:42.356652 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.356634    2581 scope.go:117] "RemoveContainer" containerID="ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec"
Sep 16 19:45:42.356959 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:45:42.356927    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec\": container with ID starting with ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec not found: ID does not exist" containerID="ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec"
Sep 16 19:45:42.357064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.356972    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec"} err="failed to get container status \"ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec\": rpc error: code = NotFound desc = could not find container \"ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec\": container with ID starting with ab5e74875cb187cac3141d8b3c4ad72710b750eb0add4be7f2173eefd50e30ec not found: ID does not exist"
Sep 16 19:45:42.373559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.373530    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"]
Sep 16 19:45:42.379411 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:42.379382    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58plflw"]
Sep 16 19:45:43.223509 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:43.223476    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" path="/var/lib/kubelet/pods/52af8ec5-4f53-48cd-91f2-a8686f10331c/volumes"
Sep 16 19:45:52.263416 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:52.263381    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x_e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f/storage-initializer/0.log"
Sep 16 19:45:52.272922 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:52.272897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x_e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f/main/0.log"
Sep 16 19:45:56.087104 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.087065    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"]
Sep 16 19:45:56.087722 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.087429    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main"
Sep 16 19:45:56.087722 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.087450    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main"
Sep 16 19:45:56.087722 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.087516    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="52af8ec5-4f53-48cd-91f2-a8686f10331c" containerName="main"
Sep 16 19:45:56.090845 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.090823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.094929 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.094904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-qmm49\""
Sep 16 19:45:56.095293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.095275    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:45:56.095572 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.095554    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:45:56.096358 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.096333    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:45:56.096458 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.096383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-qf6nc\""
Sep 16 19:45:56.106005 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.105977    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"]
Sep 16 19:45:56.212624 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.212585    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa05b51b-7652-46da-aeb5-90b14c9c6742-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.212624 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.212621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwqqt\" (UniqueName: \"kubernetes.io/projected/fa05b51b-7652-46da-aeb5-90b14c9c6742-kube-api-access-zwqqt\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.313160 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.313126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa05b51b-7652-46da-aeb5-90b14c9c6742-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.313160 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.313162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwqqt\" (UniqueName: \"kubernetes.io/projected/fa05b51b-7652-46da-aeb5-90b14c9c6742-kube-api-access-zwqqt\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.315618 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.315592    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa05b51b-7652-46da-aeb5-90b14c9c6742-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.322226 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.322206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwqqt\" (UniqueName: \"kubernetes.io/projected/fa05b51b-7652-46da-aeb5-90b14c9c6742-kube-api-access-zwqqt\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.401152 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.401123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:56.678281 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:56.678248    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"]
Sep 16 19:45:56.679137 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:45:56.679114    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa05b51b_7652_46da_aeb5_90b14c9c6742.slice/crio-17131dedbc225dd374dc1f42f40dadef6a72c77b0ae05f187c28cad0d5bd87d1 WatchSource:0}: Error finding container 17131dedbc225dd374dc1f42f40dadef6a72c77b0ae05f187c28cad0d5bd87d1: Status 404 returned error can't find the container with id 17131dedbc225dd374dc1f42f40dadef6a72c77b0ae05f187c28cad0d5bd87d1
Sep 16 19:45:57.427267 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:57.427231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" event={"ID":"fa05b51b-7652-46da-aeb5-90b14c9c6742","Type":"ContainerStarted","Data":"414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a"}
Sep 16 19:45:57.427662 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:57.427284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" event={"ID":"fa05b51b-7652-46da-aeb5-90b14c9c6742","Type":"ContainerStarted","Data":"17131dedbc225dd374dc1f42f40dadef6a72c77b0ae05f187c28cad0d5bd87d1"}
Sep 16 19:45:57.427662 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:57.427331    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:45:57.450090 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:45:57.450035    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podStartSLOduration=1.450021511 podStartE2EDuration="1.450021511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:45:57.448401821 +0000 UTC m=+3658.752418788" watchObservedRunningTime="2026-09-16 19:45:57.450021511 +0000 UTC m=+3658.754038494"
Sep 16 19:46:12.546053 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:12.546017    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"]
Sep 16 19:46:27.568118 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:27.568022    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main" containerID="cri-o://d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5" gracePeriod=45
Sep 16 19:46:28.214452 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.214425    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:46:28.267151 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267076    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-cabundle-cert\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267151 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267122    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h2l6\" (UniqueName: \"kubernetes.io/projected/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kube-api-access-4h2l6\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tmp-dir\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-dshm\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267192    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tls-certs\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267209    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-home\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-model-cache\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kserve-provision-location\") pod \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\" (UID: \"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f\") "
Sep 16 19:46:28.267672 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:28.267754 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267720    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-home" (OuterVolumeSpecName: "home") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:28.268023 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.267993    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:46:28.269502 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.269461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kube-api-access-4h2l6" (OuterVolumeSpecName: "kube-api-access-4h2l6") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "kube-api-access-4h2l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:46:28.269603 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.269495    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:46:28.269603 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.269571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-dshm" (OuterVolumeSpecName: "dshm") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:28.284693 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.284664    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:28.292107 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.292082    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" (UID: "e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:46:28.374087 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374030    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374237 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374216    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374237 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374228    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374239    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h2l6\" (UniqueName: \"kubernetes.io/projected/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-kube-api-access-4h2l6\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374250    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374259    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374266    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.374313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.374274    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:46:28.435367 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.435338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:46:28.581962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.581878    2581 generic.go:356] "Generic (PLEG): container finished" podID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerID="d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5" exitCode=0
Sep 16 19:46:28.581962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.581912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" event={"ID":"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f","Type":"ContainerDied","Data":"d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5"}
Sep 16 19:46:28.581962 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.581958    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"
Sep 16 19:46:28.582564 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.581984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x" event={"ID":"e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f","Type":"ContainerDied","Data":"644de9db0ece531723a9702db6a9073ae70cb0bfa1453a827d472c3943c3ce0f"}
Sep 16 19:46:28.582564 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.582015    2581 scope.go:117] "RemoveContainer" containerID="d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5"
Sep 16 19:46:28.591380 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.591359    2581 scope.go:117] "RemoveContainer" containerID="82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd"
Sep 16 19:46:28.599253 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.599192    2581 scope.go:117] "RemoveContainer" containerID="d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5"
Sep 16 19:46:28.599451 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:46:28.599432    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5\": container with ID starting with d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5 not found: ID does not exist" containerID="d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5"
Sep 16 19:46:28.599501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.599460    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5"} err="failed to get container status \"d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5\": rpc error: code = NotFound desc = could not find container \"d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5\": container with ID starting with d84e7e21639af724bac1f46ef55ec939e097a6fc73eec22b6a5c070b8c6c35e5 not found: ID does not exist"
Sep 16 19:46:28.599501 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.599476    2581 scope.go:117] "RemoveContainer" containerID="82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd"
Sep 16 19:46:28.599735 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:46:28.599717    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd\": container with ID starting with 82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd not found: ID does not exist" containerID="82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd"
Sep 16 19:46:28.599788 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.599738    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd"} err="failed to get container status \"82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd\": rpc error: code = NotFound desc = could not find container \"82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd\": container with ID starting with 82a8a621d41d425fe5ab1a30731aa2f195aa61155c73f23a346cfaa1939d42dd not found: ID does not exist"
Sep 16 19:46:28.606056 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.606036    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"]
Sep 16 19:46:28.612454 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:28.612429    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-85cfdcbfb4-9qs7x"]
Sep 16 19:46:29.221513 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:29.221480    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" path="/var/lib/kubelet/pods/e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f/volumes"
Sep 16 19:46:41.643953 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.643918    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"]
Sep 16 19:46:41.644400 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.644380    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="storage-initializer"
Sep 16 19:46:41.644445 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.644405    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="storage-initializer"
Sep 16 19:46:41.644445 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.644430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main"
Sep 16 19:46:41.644445 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.644440    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main"
Sep 16 19:46:41.644551 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.644524    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e50e09fc-b2d4-48b8-8c36-f7e53c3d9f7f" containerName="main"
Sep 16 19:46:41.653889 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.653838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.657602 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.657569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"seaweedfs-s3-creds\""
Sep 16 19:46:41.658069 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.658048    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Sep 16 19:46:41.658069 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.658059    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"kube-root-ca.crt\""
Sep 16 19:46:41.658313 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.658298    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"default-dockercfg-lcjbm\""
Sep 16 19:46:41.658763 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.658747    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:46:41.659417 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.659396    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-bbeb2681\"/\"openshift-service-ca.crt\""
Sep 16 19:46:41.676585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.676564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"]
Sep 16 19:46:41.775897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.775852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-home\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.775907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.775965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898c9cb4-a359-4574-b46c-c31ca0c285ed-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.775981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898c9cb4-a359-4574-b46c-c31ca0c285ed-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.776002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xh2n\" (UniqueName: \"kubernetes.io/projected/898c9cb4-a359-4574-b46c-c31ca0c285ed-kube-api-access-9xh2n\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.776089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-dshm\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.776142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.776262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.776173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.876740 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.876704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-home\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.876956 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.876746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.876956 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.876778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898c9cb4-a359-4574-b46c-c31ca0c285ed-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.876956 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.876903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898c9cb4-a359-4574-b46c-c31ca0c285ed-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.876956 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.876944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xh2n\" (UniqueName: \"kubernetes.io/projected/898c9cb4-a359-4574-b46c-c31ca0c285ed-kube-api-access-9xh2n\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877194 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.876980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-dshm\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877194 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877001    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877194 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-home\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877248    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877458 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877522 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-model-cache\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.877613 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.877589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898c9cb4-a359-4574-b46c-c31ca0c285ed-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.879166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.879147    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-dshm\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.879434 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.879416    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898c9cb4-a359-4574-b46c-c31ca0c285ed-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.887403 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.887381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xh2n\" (UniqueName: \"kubernetes.io/projected/898c9cb4-a359-4574-b46c-c31ca0c285ed-kube-api-access-9xh2n\") pod \"lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") " pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:41.964305 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:41.964286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:42.247789 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:42.247759    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"]
Sep 16 19:46:42.249260 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:46:42.249230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898c9cb4_a359_4574_b46c_c31ca0c285ed.slice/crio-d06e92f25eb36e273172e9458ffe2447d2cc77c62ebb8266ec6cab9c51abec57 WatchSource:0}: Error finding container d06e92f25eb36e273172e9458ffe2447d2cc77c62ebb8266ec6cab9c51abec57: Status 404 returned error can't find the container with id d06e92f25eb36e273172e9458ffe2447d2cc77c62ebb8266ec6cab9c51abec57
Sep 16 19:46:42.662084 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:42.662040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" event={"ID":"898c9cb4-a359-4574-b46c-c31ca0c285ed","Type":"ContainerStarted","Data":"7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec"}
Sep 16 19:46:42.662551 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:42.662114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" event={"ID":"898c9cb4-a359-4574-b46c-c31ca0c285ed","Type":"ContainerStarted","Data":"d06e92f25eb36e273172e9458ffe2447d2cc77c62ebb8266ec6cab9c51abec57"}
Sep 16 19:46:45.678745 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:45.678709    2581 generic.go:356] "Generic (PLEG): container finished" podID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerID="7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec" exitCode=0
Sep 16 19:46:45.679169 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:45.678782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" event={"ID":"898c9cb4-a359-4574-b46c-c31ca0c285ed","Type":"ContainerDied","Data":"7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec"}
Sep 16 19:46:46.686275 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:46.686240    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" event={"ID":"898c9cb4-a359-4574-b46c-c31ca0c285ed","Type":"ContainerStarted","Data":"632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d"}
Sep 16 19:46:46.710271 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:46.710169    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podStartSLOduration=2.773877748 podStartE2EDuration="5.710154184s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.936276436s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:46.708673865 +0000 UTC m=+3708.012690842" watchObservedRunningTime="2026-09-16 19:46:46.710154184 +0000 UTC m=+3708.014171150"
Sep 16 19:46:46.965269 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:46.965182    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:51.965306 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:51.965264    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:46:51.966771 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:46:51.966747    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:47:01.965042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:47:01.964994    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:47:11.965136 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:47:11.965088    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:47:21.964994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:47:21.964950    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:47:31.965197 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:47:31.965146    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:47:41.965251 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:47:41.965207    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:47:51.965305 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:47:51.965212    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" probeResult="failure" output="Get \"https://10.132.0.85:8000/health\": dial tcp 10.132.0.85:8000: connect: connection refused"
Sep 16 19:48:01.975337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:01.975307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:48:01.983218 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:01.983177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:48:28.910892 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:28.910836    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp_898c9cb4-a359-4574-b46c-c31ca0c285ed/storage-initializer/0.log"
Sep 16 19:48:28.920481 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:28.920451    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-bbeb2681_lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp_898c9cb4-a359-4574-b46c-c31ca0c285ed/main/0.log"
Sep 16 19:48:30.039738 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:30.039704    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"]
Sep 16 19:48:45.063129 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.063074    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main" containerID="cri-o://632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d" gracePeriod=45
Sep 16 19:48:45.715518 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.715490    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:48:45.835623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835524    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-tmp-dir\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898c9cb4-a359-4574-b46c-c31ca0c285ed-cabundle-cert\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835623 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-home\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-kserve-provision-location\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835683    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xh2n\" (UniqueName: \"kubernetes.io/projected/898c9cb4-a359-4574-b46c-c31ca0c285ed-kube-api-access-9xh2n\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835702    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-dshm\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835751    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-model-cache\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.835954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.835794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898c9cb4-a359-4574-b46c-c31ca0c285ed-tls-certs\") pod \"898c9cb4-a359-4574-b46c-c31ca0c285ed\" (UID: \"898c9cb4-a359-4574-b46c-c31ca0c285ed\") "
Sep 16 19:48:45.836216 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.836007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898c9cb4-a359-4574-b46c-c31ca0c285ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:48:45.836216 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.836165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:45.836404 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.836380    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-home" (OuterVolumeSpecName: "home") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:45.838591 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.838561    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898c9cb4-a359-4574-b46c-c31ca0c285ed-kube-api-access-9xh2n" (OuterVolumeSpecName: "kube-api-access-9xh2n") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "kube-api-access-9xh2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:45.838720 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.838589    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898c9cb4-a359-4574-b46c-c31ca0c285ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:45.838720 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.838599    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-dshm" (OuterVolumeSpecName: "dshm") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:45.848066 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.848039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:45.862789 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.862761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "898c9cb4-a359-4574-b46c-c31ca0c285ed" (UID: "898c9cb4-a359-4574-b46c-c31ca0c285ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:48:45.937310 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937274    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937310 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937304    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/898c9cb4-a359-4574-b46c-c31ca0c285ed-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937310 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937313    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937310 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937321    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937584 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937330    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xh2n\" (UniqueName: \"kubernetes.io/projected/898c9cb4-a359-4574-b46c-c31ca0c285ed-kube-api-access-9xh2n\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937584 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937338    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937584 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937347    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898c9cb4-a359-4574-b46c-c31ca0c285ed-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:45.937584 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:45.937355    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/898c9cb4-a359-4574-b46c-c31ca0c285ed-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:48:46.313515 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.313475    2581 generic.go:356] "Generic (PLEG): container finished" podID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerID="632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d" exitCode=0
Sep 16 19:48:46.313944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.313535    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"
Sep 16 19:48:46.313944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.313558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" event={"ID":"898c9cb4-a359-4574-b46c-c31ca0c285ed","Type":"ContainerDied","Data":"632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d"}
Sep 16 19:48:46.313944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.313616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp" event={"ID":"898c9cb4-a359-4574-b46c-c31ca0c285ed","Type":"ContainerDied","Data":"d06e92f25eb36e273172e9458ffe2447d2cc77c62ebb8266ec6cab9c51abec57"}
Sep 16 19:48:46.313944 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.313642    2581 scope.go:117] "RemoveContainer" containerID="632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d"
Sep 16 19:48:46.323078 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.323055    2581 scope.go:117] "RemoveContainer" containerID="7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec"
Sep 16 19:48:46.332236 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.332207    2581 scope.go:117] "RemoveContainer" containerID="632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d"
Sep 16 19:48:46.332590 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:48:46.332562    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d\": container with ID starting with 632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d not found: ID does not exist" containerID="632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d"
Sep 16 19:48:46.332681 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.332603    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d"} err="failed to get container status \"632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d\": rpc error: code = NotFound desc = could not find container \"632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d\": container with ID starting with 632375150d5a950153e3779925c3d6aabe97bd69a30110ce32a8b895e3eb722d not found: ID does not exist"
Sep 16 19:48:46.332681 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.332629    2581 scope.go:117] "RemoveContainer" containerID="7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec"
Sep 16 19:48:46.332973 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:48:46.332951    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec\": container with ID starting with 7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec not found: ID does not exist" containerID="7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec"
Sep 16 19:48:46.333048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.332982    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec"} err="failed to get container status \"7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec\": rpc error: code = NotFound desc = could not find container \"7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec\": container with ID starting with 7ac9b05932fea513870b6c880022af79317427153af9197ed71ab830d84d04ec not found: ID does not exist"
Sep 16 19:48:46.338536 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.338515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"]
Sep 16 19:48:46.342493 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:46.342467    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-bbeb2681/lora-multiple-adapters-test-kserve-6cf446ffff-5xpgp"]
Sep 16 19:48:47.221669 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:47.221631    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" path="/var/lib/kubelet/pods/898c9cb4-a359-4574-b46c-c31ca0c285ed/volumes"
Sep 16 19:48:53.443333 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.443291    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"]
Sep 16 19:48:53.443811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.443755    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="storage-initializer"
Sep 16 19:48:53.443811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.443773    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="storage-initializer"
Sep 16 19:48:53.443811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.443784    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main"
Sep 16 19:48:53.443811 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.443792    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main"
Sep 16 19:48:53.444086 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.443931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="898c9cb4-a359-4574-b46c-c31ca0c285ed" containerName="main"
Sep 16 19:48:53.446887 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.446845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:53.449586 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.449561    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:48:53.450653 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.450632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:48:53.450653 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.450644    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:48:53.450798 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.450644    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-wfptz\""
Sep 16 19:48:53.454580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.454553    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"]
Sep 16 19:48:53.606225 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.606184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6wk\" (UniqueName: \"kubernetes.io/projected/84d66658-380d-4a59-920a-49ba26c142a1-kube-api-access-2j6wk\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:53.707340 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.707240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6wk\" (UniqueName: \"kubernetes.io/projected/84d66658-380d-4a59-920a-49ba26c142a1-kube-api-access-2j6wk\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:53.716329 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:53.716305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6wk\" (UniqueName: \"kubernetes.io/projected/84d66658-380d-4a59-920a-49ba26c142a1-kube-api-access-2j6wk\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:55.926491 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:55.926454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:55.932286 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:55.932264    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.027478 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.027437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.027666 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.027580    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.031103 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.031078    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") DevicePath \"csi-a82bc1de3a7e9d1eb3ba75d27938354f2dec07e14c27f767b8997ffe1f881e30\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.246671 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.246640    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a7e01917307d7e1bc93c550a15ed24e39432964f889a5675dcce93e3c3daace0/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.259172 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.259145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"e2e-pvc-lora-download-cb8xg\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.457144 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.457112    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:48:56.729182 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:56.729155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"]
Sep 16 19:48:56.730664 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:48:56.730638    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84d66658_380d_4a59_920a_49ba26c142a1.slice/crio-bd84576bfda045959fb854211363d3419878eb67ac1c506f1bb4182cfdcfdc39 WatchSource:0}: Error finding container bd84576bfda045959fb854211363d3419878eb67ac1c506f1bb4182cfdcfdc39: Status 404 returned error can't find the container with id bd84576bfda045959fb854211363d3419878eb67ac1c506f1bb4182cfdcfdc39
Sep 16 19:48:57.374479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:57.374439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg" event={"ID":"84d66658-380d-4a59-920a-49ba26c142a1","Type":"ContainerStarted","Data":"ecfa2aedb7e6f312cb66cf78a0f7b8a5984abbba3f5c124560ec8b9210abd42a"}
Sep 16 19:48:57.374479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:57.374484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg" event={"ID":"84d66658-380d-4a59-920a-49ba26c142a1","Type":"ContainerStarted","Data":"bd84576bfda045959fb854211363d3419878eb67ac1c506f1bb4182cfdcfdc39"}
Sep 16 19:48:57.394087 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:57.394031    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg" podStartSLOduration=7.394014756 podStartE2EDuration="7.394014756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:48:57.393364283 +0000 UTC m=+3838.697381250" watchObservedRunningTime="2026-09-16 19:48:57.394014756 +0000 UTC m=+3838.698031724"
Sep 16 19:48:59.386840 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:59.386795    2581 generic.go:356] "Generic (PLEG): container finished" podID="84d66658-380d-4a59-920a-49ba26c142a1" containerID="ecfa2aedb7e6f312cb66cf78a0f7b8a5984abbba3f5c124560ec8b9210abd42a" exitCode=0
Sep 16 19:48:59.387300 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:48:59.386889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg" event={"ID":"84d66658-380d-4a59-920a-49ba26c142a1","Type":"ContainerDied","Data":"ecfa2aedb7e6f312cb66cf78a0f7b8a5984abbba3f5c124560ec8b9210abd42a"}
Sep 16 19:49:00.529917 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.529889    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:49:00.668996 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.668963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j6wk\" (UniqueName: \"kubernetes.io/projected/84d66658-380d-4a59-920a-49ba26c142a1-kube-api-access-2j6wk\") pod \"84d66658-380d-4a59-920a-49ba26c142a1\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") "
Sep 16 19:49:00.669195 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.669123    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"84d66658-380d-4a59-920a-49ba26c142a1\" (UID: \"84d66658-380d-4a59-920a-49ba26c142a1\") "
Sep 16 19:49:00.671355 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.671326    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d66658-380d-4a59-920a-49ba26c142a1-kube-api-access-2j6wk" (OuterVolumeSpecName: "kube-api-access-2j6wk") pod "84d66658-380d-4a59-920a-49ba26c142a1" (UID: "84d66658-380d-4a59-920a-49ba26c142a1"). InnerVolumeSpecName "kube-api-access-2j6wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:49:00.672991 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.672968    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de" (OuterVolumeSpecName: "model-storage") pod "84d66658-380d-4a59-920a-49ba26c142a1" (UID: "84d66658-380d-4a59-920a-49ba26c142a1"). InnerVolumeSpecName "pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:49:00.770479 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.770433    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6wk\" (UniqueName: \"kubernetes.io/projected/84d66658-380d-4a59-920a-49ba26c142a1-kube-api-access-2j6wk\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:49:00.770676 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:00.770492    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") on node \"ip-10-0-133-194.ec2.internal\" "
Sep 16 19:49:01.398378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:01.398343    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"
Sep 16 19:49:01.398378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:01.398368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg" event={"ID":"84d66658-380d-4a59-920a-49ba26c142a1","Type":"ContainerDied","Data":"bd84576bfda045959fb854211363d3419878eb67ac1c506f1bb4182cfdcfdc39"}
Sep 16 19:49:01.398585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:01.398400    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd84576bfda045959fb854211363d3419878eb67ac1c506f1bb4182cfdcfdc39"
Sep 16 19:49:01.742730 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:01.742703    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de") on node "ip-10-0-133-194.ec2.internal"
Sep 16 19:49:01.780203 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:01.780170    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"csi-a82bc1de3a7e9d1eb3ba75d27938354f2dec07e14c27f767b8997ffe1f881e30\""
Sep 16 19:49:17.923209 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.923176    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"]
Sep 16 19:49:17.923678 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.923657    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d66658-380d-4a59-920a-49ba26c142a1" containerName="storage-initializer"
Sep 16 19:49:17.923754 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.923683    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d66658-380d-4a59-920a-49ba26c142a1" containerName="storage-initializer"
Sep 16 19:49:17.923812 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.923772    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d66658-380d-4a59-920a-49ba26c142a1" containerName="storage-initializer"
Sep 16 19:49:17.926769 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.926747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:17.929759 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.929736    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:49:17.929759 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.929750    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 16 19:49:17.929982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.929756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 16 19:49:17.929982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.929809    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 16 19:49:17.929982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.929847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-wfptz\""
Sep 16 19:49:17.930132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.929997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 16 19:49:17.939776 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:17.939753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"]
Sep 16 19:49:18.004407 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004372    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004575 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmglb\" (UniqueName: \"kubernetes.io/projected/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kube-api-access-rmglb\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004575 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004575 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-home\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004688 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004575    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004688 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004688 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004649    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.004688 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.004670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.105733 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.105701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.105936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.105744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-home\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.105936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.105765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.105936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.105809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.105936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.105832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.105936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.105872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmglb\" (UniqueName: \"kubernetes.io/projected/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kube-api-access-rmglb\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106219 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106206    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-home\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.106624 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.106602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.108104 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.108077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.108541 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.108524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:18.115482 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:18.115463    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmglb\" (UniqueName: \"kubernetes.io/projected/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kube-api-access-rmglb\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.478599 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.478566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.485344 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.485321    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.579348 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.579306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.579517 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.579442    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.583088 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.583066    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") DevicePath \"csi-a82bc1de3a7e9d1eb3ba75d27938354f2dec07e14c27f767b8997ffe1f881e30\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.672687 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.672657    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a7e01917307d7e1bc93c550a15ed24e39432964f889a5675dcce93e3c3daace0/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.691998 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.691970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"lora-pvc-single-adapter-test-kserve-6779747d78-sffw6\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:26.937534 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:26.937502    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:27.219134 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:49:27.219101    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601e7323_5edf_4e7d_855f_5fc8bd1dc97a.slice/crio-2ea9ebafb92089f0d532f45d60269b811a5c2a190ed1a325cf0236566fad3dd2 WatchSource:0}: Error finding container 2ea9ebafb92089f0d532f45d60269b811a5c2a190ed1a325cf0236566fad3dd2: Status 404 returned error can't find the container with id 2ea9ebafb92089f0d532f45d60269b811a5c2a190ed1a325cf0236566fad3dd2
Sep 16 19:49:27.222110 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:27.222085    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"]
Sep 16 19:49:27.523556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:27.523460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" event={"ID":"601e7323-5edf-4e7d-855f-5fc8bd1dc97a","Type":"ContainerStarted","Data":"c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668"}
Sep 16 19:49:27.523556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:27.523522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" event={"ID":"601e7323-5edf-4e7d-855f-5fc8bd1dc97a","Type":"ContainerStarted","Data":"2ea9ebafb92089f0d532f45d60269b811a5c2a190ed1a325cf0236566fad3dd2"}
Sep 16 19:49:30.540833 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:30.540800    2581 generic.go:356] "Generic (PLEG): container finished" podID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerID="c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668" exitCode=0
Sep 16 19:49:30.541227 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:30.540899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" event={"ID":"601e7323-5edf-4e7d-855f-5fc8bd1dc97a","Type":"ContainerDied","Data":"c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668"}
Sep 16 19:49:31.549218 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:31.549182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" event={"ID":"601e7323-5edf-4e7d-855f-5fc8bd1dc97a","Type":"ContainerStarted","Data":"767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1"}
Sep 16 19:49:31.575970 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:31.575911    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podStartSLOduration=12.243736741 podStartE2EDuration="14.575878188s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.332141447s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:31.572396161 +0000 UTC m=+3872.876413128" watchObservedRunningTime="2026-09-16 19:49:31.575878188 +0000 UTC m=+3872.879895152"
Sep 16 19:49:31.938518 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:31.938479    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:36.937919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:36.937880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:49:36.939573 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:36.939533    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:49:46.938223 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:46.938175    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:49:56.938264 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:56.938220    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:49:59.558155 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:59.558123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:49:59.561408 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:59.561381    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:49:59.564580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:59.564556    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:49:59.567701 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:49:59.567683    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:50:06.938913 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:06.938852    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:50:15.189549 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:15.189508    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"]
Sep 16 19:50:16.938386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:16.938343    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:50:18.434927 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:18.434890    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:26.938875 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:26.938815    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:50:28.434719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:28.434672    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:30.208182 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:30.208136    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main" containerID="cri-o://414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a" gracePeriod=45
Sep 16 19:50:36.938266 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:36.938224    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:50:38.435048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:38.435010    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:38.435470 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:38.435110    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:50:46.948586 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:46.948547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:50:46.956637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:46.956604    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:50:48.435513 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:48.435470    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:50:58.435133 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:50:58.435034    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:00.473567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.473509    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:51:00.587390 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.587350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwqqt\" (UniqueName: \"kubernetes.io/projected/fa05b51b-7652-46da-aeb5-90b14c9c6742-kube-api-access-zwqqt\") pod \"fa05b51b-7652-46da-aeb5-90b14c9c6742\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") "
Sep 16 19:51:00.587567 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.587430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa05b51b-7652-46da-aeb5-90b14c9c6742-tls-certs\") pod \"fa05b51b-7652-46da-aeb5-90b14c9c6742\" (UID: \"fa05b51b-7652-46da-aeb5-90b14c9c6742\") "
Sep 16 19:51:00.589545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.589511    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa05b51b-7652-46da-aeb5-90b14c9c6742-kube-api-access-zwqqt" (OuterVolumeSpecName: "kube-api-access-zwqqt") pod "fa05b51b-7652-46da-aeb5-90b14c9c6742" (UID: "fa05b51b-7652-46da-aeb5-90b14c9c6742"). InnerVolumeSpecName "kube-api-access-zwqqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:51:00.589545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.589524    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa05b51b-7652-46da-aeb5-90b14c9c6742-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa05b51b-7652-46da-aeb5-90b14c9c6742" (UID: "fa05b51b-7652-46da-aeb5-90b14c9c6742"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:51:00.688057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.688022    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwqqt\" (UniqueName: \"kubernetes.io/projected/fa05b51b-7652-46da-aeb5-90b14c9c6742-kube-api-access-zwqqt\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:00.688057 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:00.688050    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa05b51b-7652-46da-aeb5-90b14c9c6742-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:51:01.037153 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.037111    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerID="414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a" exitCode=0
Sep 16 19:51:01.037153 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.037152    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"
Sep 16 19:51:01.037359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.037174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" event={"ID":"fa05b51b-7652-46da-aeb5-90b14c9c6742","Type":"ContainerDied","Data":"414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a"}
Sep 16 19:51:01.037359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.037228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c" event={"ID":"fa05b51b-7652-46da-aeb5-90b14c9c6742","Type":"ContainerDied","Data":"17131dedbc225dd374dc1f42f40dadef6a72c77b0ae05f187c28cad0d5bd87d1"}
Sep 16 19:51:01.037359 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.037260    2581 scope.go:117] "RemoveContainer" containerID="414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a"
Sep 16 19:51:01.046728 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.046710    2581 scope.go:117] "RemoveContainer" containerID="414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a"
Sep 16 19:51:01.047057 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:51:01.047036    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a\": container with ID starting with 414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a not found: ID does not exist" containerID="414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a"
Sep 16 19:51:01.047116 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.047065    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a"} err="failed to get container status \"414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a\": rpc error: code = NotFound desc = could not find container \"414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a\": container with ID starting with 414d6c2b1aeb7dfb2c8aa1be1aafd119d9b0011b11b9eef31a0a0aeb6ea8670a not found: ID does not exist"
Sep 16 19:51:01.061377 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.061342    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"]
Sep 16 19:51:01.065823 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.065800    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-mxk8c"]
Sep 16 19:51:01.221845 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:01.221811    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" path="/var/lib/kubelet/pods/fa05b51b-7652-46da-aeb5-90b14c9c6742/volumes"
Sep 16 19:51:11.789977 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:11.789945    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6779747d78-sffw6_601e7323-5edf-4e7d-855f-5fc8bd1dc97a/storage-initializer/0.log"
Sep 16 19:51:11.803770 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:11.803740    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6779747d78-sffw6_601e7323-5edf-4e7d-855f-5fc8bd1dc97a/main/0.log"
Sep 16 19:51:24.895322 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.895286    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"]
Sep 16 19:51:24.895810 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.895663    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main"
Sep 16 19:51:24.895810 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.895676    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main"
Sep 16 19:51:24.895810 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.895741    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa05b51b-7652-46da-aeb5-90b14c9c6742" containerName="main"
Sep 16 19:51:24.898699 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.898679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:24.901460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.901404    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:51:24.901460 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.901412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:51:24.901648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.901480    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-5lvhp\""
Sep 16 19:51:24.901648 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.901501    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:51:24.901897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.901843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 19:51:24.902584 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.902561    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:51:24.914583 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:24.914551    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"]
Sep 16 19:51:25.004490 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a62af2-4dc8-473a-8f99-40680647b969-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj88g\" (UniqueName: \"kubernetes.io/projected/d5a62af2-4dc8-473a-8f99-40680647b969-kube-api-access-pj88g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a62af2-4dc8-473a-8f99-40680647b969-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.004820 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.004813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.105796 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.105753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a62af2-4dc8-473a-8f99-40680647b969-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106001 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.105809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106001 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.105852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.105993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj88g\" (UniqueName: \"kubernetes.io/projected/d5a62af2-4dc8-473a-8f99-40680647b969-kube-api-access-pj88g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a62af2-4dc8-473a-8f99-40680647b969-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106319 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106319 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106430 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106369    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106485 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106544 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106518    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.106631 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.106615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a62af2-4dc8-473a-8f99-40680647b969-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.108485 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.108467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.108840 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.108823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a62af2-4dc8-473a-8f99-40680647b969-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.114352 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.114326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj88g\" (UniqueName: \"kubernetes.io/projected/d5a62af2-4dc8-473a-8f99-40680647b969-kube-api-access-pj88g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.210164 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.210127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:25.497320 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:25.497290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"]
Sep 16 19:51:25.498808 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:51:25.498784    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a62af2_4dc8_473a_8f99_40680647b969.slice/crio-e3b0864ea7b47e93344ebc6359d841b8db203e456f75397e7d474e3d61d11e0b WatchSource:0}: Error finding container e3b0864ea7b47e93344ebc6359d841b8db203e456f75397e7d474e3d61d11e0b: Status 404 returned error can't find the container with id e3b0864ea7b47e93344ebc6359d841b8db203e456f75397e7d474e3d61d11e0b
Sep 16 19:51:26.175040 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:26.174992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" event={"ID":"d5a62af2-4dc8-473a-8f99-40680647b969","Type":"ContainerStarted","Data":"1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2"}
Sep 16 19:51:26.175492 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:26.175058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" event={"ID":"d5a62af2-4dc8-473a-8f99-40680647b969","Type":"ContainerStarted","Data":"e3b0864ea7b47e93344ebc6359d841b8db203e456f75397e7d474e3d61d11e0b"}
Sep 16 19:51:28.187111 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:28.187076    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5a62af2-4dc8-473a-8f99-40680647b969" containerID="1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2" exitCode=0
Sep 16 19:51:28.187508 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:28.187140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" event={"ID":"d5a62af2-4dc8-473a-8f99-40680647b969","Type":"ContainerDied","Data":"1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2"}
Sep 16 19:51:29.194842 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:29.194803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" event={"ID":"d5a62af2-4dc8-473a-8f99-40680647b969","Type":"ContainerStarted","Data":"484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e"}
Sep 16 19:51:29.243080 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:29.242979    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podStartSLOduration=2.932305983 podStartE2EDuration="5.242955938s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.310649955s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:51:29.235457681 +0000 UTC m=+3990.539474649" watchObservedRunningTime="2026-09-16 19:51:29.242955938 +0000 UTC m=+3990.546972909"
Sep 16 19:51:35.211048 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:35.211009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:35.211512 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:35.211086    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:51:35.212743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:35.212711    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:51:45.210977 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:45.210930    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:51:55.211134 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:55.211082    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:51:56.494761 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:51:56.494723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"]
Sep 16 19:52:05.211371 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:05.211321    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:52:11.517868 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:11.517812    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" containerID="cri-o://767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1" gracePeriod=45
Sep 16 19:52:11.951743 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:11.951681    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": dial tcp 10.132.0.87:8000: connect: connection refused"
Sep 16 19:52:12.184547 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.184520    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:52:12.317058 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.316971    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317058 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-cabundle-cert\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kserve-provision-location\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tls-certs\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tmp-dir\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-model-cache\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-dshm\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317260 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmglb\" (UniqueName: \"kubernetes.io/projected/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kube-api-access-rmglb\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317581 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-home\") pod \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\" (UID: \"601e7323-5edf-4e7d-855f-5fc8bd1dc97a\") "
Sep 16 19:52:12.317581 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:52:12.317701 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317578    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.317783 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317762    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-model-cache" (OuterVolumeSpecName: "model-cache") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:12.317844 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.317820    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-home" (OuterVolumeSpecName: "home") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:12.319589 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.319548    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:52:12.320970 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.320937    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kube-api-access-rmglb" (OuterVolumeSpecName: "kube-api-access-rmglb") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "kube-api-access-rmglb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:12.322337 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.322310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-dshm" (OuterVolumeSpecName: "dshm") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:12.323053 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.323030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:52:12.328704 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.328678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:12.342809 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.342774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "601e7323-5edf-4e7d-855f-5fc8bd1dc97a" (UID: "601e7323-5edf-4e7d-855f-5fc8bd1dc97a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:52:12.418102 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418067    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418102 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418099    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418111    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418123    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418133    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418144    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmglb\" (UniqueName: \"kubernetes.io/projected/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-kube-api-access-rmglb\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418154    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/601e7323-5edf-4e7d-855f-5fc8bd1dc97a-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:12.418349 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.418189    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") on node \"ip-10-0-133-194.ec2.internal\" "
Sep 16 19:52:12.425167 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.425134    2581 generic.go:356] "Generic (PLEG): container finished" podID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerID="767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1" exitCode=0
Sep 16 19:52:12.425284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.425172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" event={"ID":"601e7323-5edf-4e7d-855f-5fc8bd1dc97a","Type":"ContainerDied","Data":"767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1"}
Sep 16 19:52:12.425284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.425215    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"
Sep 16 19:52:12.425284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.425245    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6" event={"ID":"601e7323-5edf-4e7d-855f-5fc8bd1dc97a","Type":"ContainerDied","Data":"2ea9ebafb92089f0d532f45d60269b811a5c2a190ed1a325cf0236566fad3dd2"}
Sep 16 19:52:12.425284 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.425278    2581 scope.go:117] "RemoveContainer" containerID="767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1"
Sep 16 19:52:12.428282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.428266    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de") on node "ip-10-0-133-194.ec2.internal"
Sep 16 19:52:12.434159 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.434141    2581 scope.go:117] "RemoveContainer" containerID="c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668"
Sep 16 19:52:12.442263 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.442246    2581 scope.go:117] "RemoveContainer" containerID="767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1"
Sep 16 19:52:12.442545 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:52:12.442527    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1\": container with ID starting with 767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1 not found: ID does not exist" containerID="767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1"
Sep 16 19:52:12.442590 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.442553    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1"} err="failed to get container status \"767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1\": rpc error: code = NotFound desc = could not find container \"767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1\": container with ID starting with 767250a246c60a9d968b7f2102be21369d7dc724f44c0816e188a1f303383af1 not found: ID does not exist"
Sep 16 19:52:12.442590 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.442570    2581 scope.go:117] "RemoveContainer" containerID="c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668"
Sep 16 19:52:12.442842 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:52:12.442823    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668\": container with ID starting with c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668 not found: ID does not exist" containerID="c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668"
Sep 16 19:52:12.442918 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.442848    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668"} err="failed to get container status \"c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668\": rpc error: code = NotFound desc = could not find container \"c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668\": container with ID starting with c8f5d3d0302d30c81b5fc04b2f144148fc5289df0abe0ccd07a94ffdbfd50668 not found: ID does not exist"
Sep 16 19:52:12.453425 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.453401    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"]
Sep 16 19:52:12.456989 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.456966    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6779747d78-sffw6"]
Sep 16 19:52:12.518847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:12.518811    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-7fc78ee3-60d9-4be5-8e1e-09f8a254aa28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-085a0b82a384c83de\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"csi-a82bc1de3a7e9d1eb3ba75d27938354f2dec07e14c27f767b8997ffe1f881e30\""
Sep 16 19:52:13.223135 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:13.223043    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" path="/var/lib/kubelet/pods/601e7323-5edf-4e7d-855f-5fc8bd1dc97a/volumes"
Sep 16 19:52:15.210897 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:15.210840    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:52:16.670265 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.670235    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"]
Sep 16 19:52:16.670677 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.670543    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main"
Sep 16 19:52:16.670677 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.670555    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main"
Sep 16 19:52:16.670677 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.670582    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="storage-initializer"
Sep 16 19:52:16.670677 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.670589    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="storage-initializer"
Sep 16 19:52:16.670677 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.670636    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="601e7323-5edf-4e7d-855f-5fc8bd1dc97a" containerName="main"
Sep 16 19:52:16.675054 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.675037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:16.677922 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.677899    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:52:16.679199 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.679174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:52:16.679466 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.679443    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 16 19:52:16.679755 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.679734    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-84m8j\""
Sep 16 19:52:16.682656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.682615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"]
Sep 16 19:52:16.755434 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.755399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlt9b\" (UniqueName: \"kubernetes.io/projected/58017ad4-44d6-4d33-a500-c5f361a5418a-kube-api-access-wlt9b\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:16.856823 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.856782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlt9b\" (UniqueName: \"kubernetes.io/projected/58017ad4-44d6-4d33-a500-c5f361a5418a-kube-api-access-wlt9b\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:16.865717 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:16.865692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlt9b\" (UniqueName: \"kubernetes.io/projected/58017ad4-44d6-4d33-a500-c5f361a5418a-kube-api-access-wlt9b\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:18.339683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:18.339650    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"]
Sep 16 19:52:18.345366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:18.345340    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-cb8xg"]
Sep 16 19:52:19.222556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:19.222525    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d66658-380d-4a59-920a-49ba26c142a1" path="/var/lib/kubelet/pods/84d66658-380d-4a59-920a-49ba26c142a1/volumes"
Sep 16 19:52:20.789952 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:20.789917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:20.795594 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:20.795572    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:20.891114 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:20.891074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:20.891271 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:20.891212    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:20.894559 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:20.894540    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") DevicePath \"csi-74ea00934924a51cf0b20c93618ed9fbf75e34f929e922eb1152055ac6f49d4c\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:21.543080 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:21.543042    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2eeaf6b15fe5714c15e7850371a6074195f6504732261b6b4c6f711855610d87/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:21.556945 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:21.556919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"e2e-pvc-lora-download-k27bq\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:21.786538 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:21.786416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:22.063555 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:22.063466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"]
Sep 16 19:52:22.067250 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:52:22.067220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58017ad4_44d6_4d33_a500_c5f361a5418a.slice/crio-2f1b9b40c5f4142a2e2c8afcb416a425c7ef1df225812ce82861c05873abca64 WatchSource:0}: Error finding container 2f1b9b40c5f4142a2e2c8afcb416a425c7ef1df225812ce82861c05873abca64: Status 404 returned error can't find the container with id 2f1b9b40c5f4142a2e2c8afcb416a425c7ef1df225812ce82861c05873abca64
Sep 16 19:52:22.480772 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:22.480733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq" event={"ID":"58017ad4-44d6-4d33-a500-c5f361a5418a","Type":"ContainerStarted","Data":"090acf943cf86c808d888c92227201149a86489fc590229eff7116e436dfdfb2"}
Sep 16 19:52:22.480982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:22.480784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq" event={"ID":"58017ad4-44d6-4d33-a500-c5f361a5418a","Type":"ContainerStarted","Data":"2f1b9b40c5f4142a2e2c8afcb416a425c7ef1df225812ce82861c05873abca64"}
Sep 16 19:52:22.503603 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:22.503537    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq" podStartSLOduration=9.503516225 podStartE2EDuration="9.503516225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:22.499517795 +0000 UTC m=+4043.803534762" watchObservedRunningTime="2026-09-16 19:52:22.503516225 +0000 UTC m=+4043.807533193"
Sep 16 19:52:24.494321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:24.494231    2581 generic.go:356] "Generic (PLEG): container finished" podID="58017ad4-44d6-4d33-a500-c5f361a5418a" containerID="090acf943cf86c808d888c92227201149a86489fc590229eff7116e436dfdfb2" exitCode=0
Sep 16 19:52:24.494321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:24.494303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq" event={"ID":"58017ad4-44d6-4d33-a500-c5f361a5418a","Type":"ContainerDied","Data":"090acf943cf86c808d888c92227201149a86489fc590229eff7116e436dfdfb2"}
Sep 16 19:52:25.211234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.211182    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:52:25.645150 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.645126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:25.732988 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.732950    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"58017ad4-44d6-4d33-a500-c5f361a5418a\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") "
Sep 16 19:52:25.733189 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.733032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlt9b\" (UniqueName: \"kubernetes.io/projected/58017ad4-44d6-4d33-a500-c5f361a5418a-kube-api-access-wlt9b\") pod \"58017ad4-44d6-4d33-a500-c5f361a5418a\" (UID: \"58017ad4-44d6-4d33-a500-c5f361a5418a\") "
Sep 16 19:52:25.735401 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.735370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58017ad4-44d6-4d33-a500-c5f361a5418a-kube-api-access-wlt9b" (OuterVolumeSpecName: "kube-api-access-wlt9b") pod "58017ad4-44d6-4d33-a500-c5f361a5418a" (UID: "58017ad4-44d6-4d33-a500-c5f361a5418a"). InnerVolumeSpecName "kube-api-access-wlt9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:25.735813 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.735790    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd" (OuterVolumeSpecName: "model-storage") pod "58017ad4-44d6-4d33-a500-c5f361a5418a" (UID: "58017ad4-44d6-4d33-a500-c5f361a5418a"). InnerVolumeSpecName "pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:52:25.834244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.834158    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlt9b\" (UniqueName: \"kubernetes.io/projected/58017ad4-44d6-4d33-a500-c5f361a5418a-kube-api-access-wlt9b\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:52:25.834244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:25.834204    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") on node \"ip-10-0-133-194.ec2.internal\" "
Sep 16 19:52:26.505702 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:26.505665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"
Sep 16 19:52:26.505702 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:26.505683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq" event={"ID":"58017ad4-44d6-4d33-a500-c5f361a5418a","Type":"ContainerDied","Data":"2f1b9b40c5f4142a2e2c8afcb416a425c7ef1df225812ce82861c05873abca64"}
Sep 16 19:52:26.505991 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:26.505714    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f1b9b40c5f4142a2e2c8afcb416a425c7ef1df225812ce82861c05873abca64"
Sep 16 19:52:26.804527 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:26.804506    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd") on node "ip-10-0-133-194.ec2.internal"
Sep 16 19:52:26.842764 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:26.842728    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"csi-74ea00934924a51cf0b20c93618ed9fbf75e34f929e922eb1152055ac6f49d4c\""
Sep 16 19:52:35.211497 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:35.211454    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:52:41.514283 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.514245    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"]
Sep 16 19:52:41.514695 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.514679    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58017ad4-44d6-4d33-a500-c5f361a5418a" containerName="storage-initializer"
Sep 16 19:52:41.514746 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.514697    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58017ad4-44d6-4d33-a500-c5f361a5418a" containerName="storage-initializer"
Sep 16 19:52:41.514787 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.514778    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58017ad4-44d6-4d33-a500-c5f361a5418a" containerName="storage-initializer"
Sep 16 19:52:41.518365 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.518340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.521304 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.521284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 16 19:52:41.521660 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.521637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 16 19:52:41.521773 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.521719    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-84m8j\""
Sep 16 19:52:41.521773 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.521745    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 16 19:52:41.529420 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.529390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"]
Sep 16 19:52:41.667760 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.667730    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8k7r\" (UniqueName: \"kubernetes.io/projected/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-kube-api-access-b8k7r\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.667982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.667785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.667982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.667893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.667982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.667946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.667982 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.667974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.668172 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.668002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769151 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8k7r\" (UniqueName: \"kubernetes.io/projected/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-kube-api-access-b8k7r\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769151 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769372 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769372 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769372 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769372 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769611 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769652 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.769698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.769660    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.771614 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.771591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.771975 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.771952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:41.777542 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:41.777509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8k7r\" (UniqueName: \"kubernetes.io/projected/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-kube-api-access-b8k7r\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:45.211248 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:45.211202    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:52:51.347449 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.347401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.353505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.353477    2581 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.448002 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.447968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.448188 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.448128    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.451497 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.451472    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") DevicePath \"csi-74ea00934924a51cf0b20c93618ed9fbf75e34f929e922eb1152055ac6f49d4c\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.538490 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.538461    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/2eeaf6b15fe5714c15e7850371a6074195f6504732261b6b4c6f711855610d87/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.557564 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.557540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-88jmw\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:51.731042 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:51.731008    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:52.014866 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:52.014823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"]
Sep 16 19:52:52.016081 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:52:52.016050    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c3ebf4_d33a_4141_9db4_3fa65006f4f4.slice/crio-c5d2f06615d089b6ea1524a58806cde1ab6021c4450ee803d8d7e5cfb21fa24c WatchSource:0}: Error finding container c5d2f06615d089b6ea1524a58806cde1ab6021c4450ee803d8d7e5cfb21fa24c: Status 404 returned error can't find the container with id c5d2f06615d089b6ea1524a58806cde1ab6021c4450ee803d8d7e5cfb21fa24c
Sep 16 19:52:52.646393 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:52.646352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" event={"ID":"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4","Type":"ContainerStarted","Data":"ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f"}
Sep 16 19:52:52.646788 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:52.646411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" event={"ID":"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4","Type":"ContainerStarted","Data":"c5d2f06615d089b6ea1524a58806cde1ab6021c4450ee803d8d7e5cfb21fa24c"}
Sep 16 19:52:52.667769 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:52.667712    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podStartSLOduration=11.667695636 podStartE2EDuration="11.667695636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:52.665547913 +0000 UTC m=+4073.969564880" watchObservedRunningTime="2026-09-16 19:52:52.667695636 +0000 UTC m=+4073.971712603"
Sep 16 19:52:52.731381 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:52.731347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:52:55.211574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:52:55.211526    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:53:01.732093 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:01.732042    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:53:01.733888 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:01.733836    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:53:05.211647 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:05.211600    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:53:11.732308 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:11.732262    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:53:15.211656 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:15.211608    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": dial tcp 10.132.0.88:8000: connect: connection refused"
Sep 16 19:53:21.732184 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:21.732133    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:53:25.223865 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:25.223820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:53:25.231766 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:25.231740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:53:31.731624 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:31.731576    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:53:41.731870 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:41.731814    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:53:45.061781 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:45.061745    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"]
Sep 16 19:53:51.731976 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:53:51.731884    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:54:00.084070 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.083996    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main" containerID="cri-o://484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e" gracePeriod=45
Sep 16 19:54:00.833318 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.833285    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:54:00.955096 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj88g\" (UniqueName: \"kubernetes.io/projected/d5a62af2-4dc8-473a-8f99-40680647b969-kube-api-access-pj88g\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-tmp-dir\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955166    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-kserve-provision-location\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955231    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-home\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955282 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-model-cache\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955308    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a62af2-4dc8-473a-8f99-40680647b969-tls-certs\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955330    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-dshm\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a62af2-4dc8-473a-8f99-40680647b969-cabundle-cert\") pod \"d5a62af2-4dc8-473a-8f99-40680647b969\" (UID: \"d5a62af2-4dc8-473a-8f99-40680647b969\") "
Sep 16 19:54:00.955737 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:00.956009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.955979    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a62af2-4dc8-473a-8f99-40680647b969-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:54:00.956289 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.956260    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-home" (OuterVolumeSpecName: "home") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:00.957958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.957929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a62af2-4dc8-473a-8f99-40680647b969-kube-api-access-pj88g" (OuterVolumeSpecName: "kube-api-access-pj88g") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "kube-api-access-pj88g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:54:00.957958 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.957945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-dshm" (OuterVolumeSpecName: "dshm") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:00.958120 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.958037    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a62af2-4dc8-473a-8f99-40680647b969-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:54:00.969345 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.969317    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:00.978132 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:00.978108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d5a62af2-4dc8-473a-8f99-40680647b969" (UID: "d5a62af2-4dc8-473a-8f99-40680647b969"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:54:01.025478 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.025446    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5a62af2-4dc8-473a-8f99-40680647b969" containerID="484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e" exitCode=0
Sep 16 19:54:01.025673 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.025502    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"
Sep 16 19:54:01.025673 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.025518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" event={"ID":"d5a62af2-4dc8-473a-8f99-40680647b969","Type":"ContainerDied","Data":"484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e"}
Sep 16 19:54:01.025673 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.025575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg" event={"ID":"d5a62af2-4dc8-473a-8f99-40680647b969","Type":"ContainerDied","Data":"e3b0864ea7b47e93344ebc6359d841b8db203e456f75397e7d474e3d61d11e0b"}
Sep 16 19:54:01.025673 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.025605    2581 scope.go:117] "RemoveContainer" containerID="484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e"
Sep 16 19:54:01.034931 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.034909    2581 scope.go:117] "RemoveContainer" containerID="1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2"
Sep 16 19:54:01.043698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.043680    2581 scope.go:117] "RemoveContainer" containerID="484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e"
Sep 16 19:54:01.043988 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:54:01.043966    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e\": container with ID starting with 484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e not found: ID does not exist" containerID="484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e"
Sep 16 19:54:01.044063 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.043998    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e"} err="failed to get container status \"484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e\": rpc error: code = NotFound desc = could not find container \"484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e\": container with ID starting with 484c113d3e18c94ac1bcb93b460eec965e5c469b34e6c7eb2bf7dbad48378d8e not found: ID does not exist"
Sep 16 19:54:01.044063 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.044016    2581 scope.go:117] "RemoveContainer" containerID="1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2"
Sep 16 19:54:01.044312 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:54:01.044291    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2\": container with ID starting with 1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2 not found: ID does not exist" containerID="1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2"
Sep 16 19:54:01.044377 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.044320    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2"} err="failed to get container status \"1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2\": rpc error: code = NotFound desc = could not find container \"1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2\": container with ID starting with 1ac9f78e77c76ebdb271d1a7eca5d04d22bd4f1d14f199e63dea31c72c8779d2 not found: ID does not exist"
Sep 16 19:54:01.050377 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.050356    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"]
Sep 16 19:54:01.056466 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056441    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056472    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056487    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5a62af2-4dc8-473a-8f99-40680647b969-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056501    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056514    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d5a62af2-4dc8-473a-8f99-40680647b969-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056531    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj88g\" (UniqueName: \"kubernetes.io/projected/d5a62af2-4dc8-473a-8f99-40680647b969-kube-api-access-pj88g\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056554 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056546    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.056742 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.056564    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d5a62af2-4dc8-473a-8f99-40680647b969-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:54:01.057680 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.057657    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86df9fg"]
Sep 16 19:54:01.222475 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.222395    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" path="/var/lib/kubelet/pods/d5a62af2-4dc8-473a-8f99-40680647b969/volumes"
Sep 16 19:54:01.732452 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:01.732409    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:54:11.741936 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:11.741896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:54:12.092723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:12.092640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:54:38.786406 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:38.786374    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-88jmw_c9c3ebf4-d33a-4141-9db4-3fa65006f4f4/main/0.log"
Sep 16 19:54:41.512236 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.512195    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"]
Sep 16 19:54:41.512724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.512545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="storage-initializer"
Sep 16 19:54:41.512724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.512560    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="storage-initializer"
Sep 16 19:54:41.512724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.512573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main"
Sep 16 19:54:41.512724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.512581    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main"
Sep 16 19:54:41.512724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.512635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a62af2-4dc8-473a-8f99-40680647b969" containerName="main"
Sep 16 19:54:41.516806 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.516783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.521614 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.521590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:54:41.521739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.521624    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:54:41.521739 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.521680    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:54:41.521879 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.521745    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 19:54:41.522215 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.522199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:54:41.522562 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.522545    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-rs6n4\""
Sep 16 19:54:41.530218 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.530196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"]
Sep 16 19:54:41.568699 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568669    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.568699 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.568949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.568949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.568949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bb1ff-11d6-4eac-adc0-87199e280821-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.568949 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/559bb1ff-11d6-4eac-adc0-87199e280821-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.569140 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.568968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x78tv\" (UniqueName: \"kubernetes.io/projected/559bb1ff-11d6-4eac-adc0-87199e280821-kube-api-access-x78tv\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.569140 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.569020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670317 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670317 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bb1ff-11d6-4eac-adc0-87199e280821-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670570 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/559bb1ff-11d6-4eac-adc0-87199e280821-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670570 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x78tv\" (UniqueName: \"kubernetes.io/projected/559bb1ff-11d6-4eac-adc0-87199e280821-kube-api-access-x78tv\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670570 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670740 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670740 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.670740 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.670626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.671026 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.671000    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.671166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.671063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.671166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.671136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.671293 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.671165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.671353 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.671324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/559bb1ff-11d6-4eac-adc0-87199e280821-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.672724 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.672705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.673047 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.673026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bb1ff-11d6-4eac-adc0-87199e280821-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.678832 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.678808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x78tv\" (UniqueName: \"kubernetes.io/projected/559bb1ff-11d6-4eac-adc0-87199e280821-kube-api-access-x78tv\") pod \"custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:41.828728 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:41.828648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:42.114598 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:42.114402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"]
Sep 16 19:54:42.117350 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:54:42.117321    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod559bb1ff_11d6_4eac_adc0_87199e280821.slice/crio-dea2fbbcf5155855881c86139c43d98f18c254c2e2d88932b27e9b4e7d460834 WatchSource:0}: Error finding container dea2fbbcf5155855881c86139c43d98f18c254c2e2d88932b27e9b4e7d460834: Status 404 returned error can't find the container with id dea2fbbcf5155855881c86139c43d98f18c254c2e2d88932b27e9b4e7d460834
Sep 16 19:54:42.245136 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:42.245091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" event={"ID":"559bb1ff-11d6-4eac-adc0-87199e280821","Type":"ContainerStarted","Data":"5d0b948b3ec54ce5c03f14dc5e61c7e902e84bc8195391e20036523e4621af82"}
Sep 16 19:54:42.245317 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:42.245159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" event={"ID":"559bb1ff-11d6-4eac-adc0-87199e280821","Type":"ContainerStarted","Data":"dea2fbbcf5155855881c86139c43d98f18c254c2e2d88932b27e9b4e7d460834"}
Sep 16 19:54:45.266200 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:45.266150    2581 generic.go:356] "Generic (PLEG): container finished" podID="559bb1ff-11d6-4eac-adc0-87199e280821" containerID="5d0b948b3ec54ce5c03f14dc5e61c7e902e84bc8195391e20036523e4621af82" exitCode=0
Sep 16 19:54:45.266635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:45.266252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" event={"ID":"559bb1ff-11d6-4eac-adc0-87199e280821","Type":"ContainerDied","Data":"5d0b948b3ec54ce5c03f14dc5e61c7e902e84bc8195391e20036523e4621af82"}
Sep 16 19:54:46.275690 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:46.275651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" event={"ID":"559bb1ff-11d6-4eac-adc0-87199e280821","Type":"ContainerStarted","Data":"7f1224c04d62eb0438bbe6b9246230797960e0277d72ab703d375ce6475fc090"}
Sep 16 19:54:46.300725 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:46.300667    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podStartSLOduration=2.9215787 podStartE2EDuration="5.300650392s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.379071692s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:54:46.29744022 +0000 UTC m=+4187.601457223" watchObservedRunningTime="2026-09-16 19:54:46.300650392 +0000 UTC m=+4187.604667358"
Sep 16 19:54:51.829713 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:51.829680    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:51.830212 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:51.829741    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:54:51.831529 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:51.831496    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:54:59.601774 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:59.601741    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:54:59.605076 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:59.605051    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:54:59.608234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:59.608212    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:54:59.611368 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:59.611353    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:54:59.845893 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:54:59.845841    2581 scope.go:117] "RemoveContainer" containerID="ecfa2aedb7e6f312cb66cf78a0f7b8a5984abbba3f5c124560ec8b9210abd42a"
Sep 16 19:55:00.613415 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:00.613377    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"]
Sep 16 19:55:01.829613 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:01.829567    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:55:11.829188 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:11.829134    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:55:15.636019 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:15.635955    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" containerID="cri-o://ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f" gracePeriod=45
Sep 16 19:55:16.085675 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.085635    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.90:8000/health\": dial tcp 10.132.0.90:8000: connect: connection refused"
Sep 16 19:55:16.298448 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.298421    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:55:16.372278 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372196    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tmp-dir\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372278 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-home\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372278 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-model-cache\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372475    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tls-certs\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8k7r\" (UniqueName: \"kubernetes.io/projected/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-kube-api-access-b8k7r\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-dshm\") pod \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\" (UID: \"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4\") "
Sep 16 19:55:16.372569 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:16.372837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-home" (OuterVolumeSpecName: "home") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:16.372837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372817    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:16.372837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.372835    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:16.375115 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.375074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-dshm" (OuterVolumeSpecName: "dshm") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:16.375240 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.375187    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-kube-api-access-b8k7r" (OuterVolumeSpecName: "kube-api-access-b8k7r") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "kube-api-access-b8k7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:55:16.375715 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.375688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:55:16.380544 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.380459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd" (OuterVolumeSpecName: "kserve-pvc-source") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 16 19:55:16.385178 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.385154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" (UID: "c9c3ebf4-d33a-4141-9db4-3fa65006f4f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:55:16.437667 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.437642    2581 generic.go:356] "Generic (PLEG): container finished" podID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerID="ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f" exitCode=0
Sep 16 19:55:16.437802 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.437704    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"
Sep 16 19:55:16.437802 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.437729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" event={"ID":"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4","Type":"ContainerDied","Data":"ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f"}
Sep 16 19:55:16.437802 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.437788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw" event={"ID":"c9c3ebf4-d33a-4141-9db4-3fa65006f4f4","Type":"ContainerDied","Data":"c5d2f06615d089b6ea1524a58806cde1ab6021c4450ee803d8d7e5cfb21fa24c"}
Sep 16 19:55:16.437999 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.437817    2581 scope.go:117] "RemoveContainer" containerID="ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f"
Sep 16 19:55:16.446031 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.446011    2581 scope.go:117] "RemoveContainer" containerID="ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f"
Sep 16 19:55:16.446303 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:55:16.446283    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f\": container with ID starting with ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f not found: ID does not exist" containerID="ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f"
Sep 16 19:55:16.446397 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.446313    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f"} err="failed to get container status \"ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f\": rpc error: code = NotFound desc = could not find container \"ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f\": container with ID starting with ca93c8d4e0cea809b1aa631ed955946bc7928bfcb65f1b9f37e5618718f0f76f not found: ID does not exist"
Sep 16 19:55:16.467587 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.467560    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"]
Sep 16 19:55:16.470888 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.470849    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-88jmw"]
Sep 16 19:55:16.473712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.473693    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") on node \"ip-10-0-133-194.ec2.internal\" "
Sep 16 19:55:16.473784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.473720    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:16.473784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.473737    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8k7r\" (UniqueName: \"kubernetes.io/projected/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-kube-api-access-b8k7r\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:16.473784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.473750    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:16.473784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.473763    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:55:16.507539 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.507518    2581 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd") on node "ip-10-0-133-194.ec2.internal"
Sep 16 19:55:16.575056 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:16.575025    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-0887e10d-5320-4a75-bc9a-54e97d0d38f0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0272769730d73bfbd\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"csi-74ea00934924a51cf0b20c93618ed9fbf75e34f929e922eb1152055ac6f49d4c\""
Sep 16 19:55:17.222759 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:17.222714    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" path="/var/lib/kubelet/pods/c9c3ebf4-d33a-4141-9db4-3fa65006f4f4/volumes"
Sep 16 19:55:21.829792 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:21.829700    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:55:25.014574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:25.014535    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"]
Sep 16 19:55:25.018878 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:25.018841    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-k27bq"]
Sep 16 19:55:25.221662 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:25.221623    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58017ad4-44d6-4d33-a500-c5f361a5418a" path="/var/lib/kubelet/pods/58017ad4-44d6-4d33-a500-c5f361a5418a/volumes"
Sep 16 19:55:28.920683 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.920642    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"]
Sep 16 19:55:28.921188 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.921167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main"
Sep 16 19:55:28.921253 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.921192    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main"
Sep 16 19:55:28.921310 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.921299    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c3ebf4-d33a-4141-9db4-3fa65006f4f4" containerName="main"
Sep 16 19:55:28.924915 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.924896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:28.927577 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.927553    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 16 19:55:28.927706 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.927607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:55:28.927706 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.927553    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:55:28.927834 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.927751    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qhcph\""
Sep 16 19:55:28.927834 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.927810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:55:28.928009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.927981    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:55:28.937124 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:28.937099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"]
Sep 16 19:55:29.081462 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-cabundle-cert\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tls-certs\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-model-cache\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6qqz\" (UniqueName: \"kubernetes.io/projected/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kube-api-access-k6qqz\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kserve-provision-location\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-home\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081649 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tmp-dir\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.081898 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.081671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-dshm\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.179849 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.179767    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"]
Sep 16 19:55:29.183006 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.182976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-cabundle-cert\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183143 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tls-certs\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183143 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-model-cache\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183143 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6qqz\" (UniqueName: \"kubernetes.io/projected/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kube-api-access-k6qqz\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183320 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kserve-provision-location\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183320 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-home\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183320 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tmp-dir\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183320 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-dshm\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183538 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kserve-provision-location\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183598 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-home\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183756 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tmp-dir\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.183838 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.183929 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.183803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-cabundle-cert\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.184051 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.184029    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-model-cache\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.185954 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.185930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-dshm\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.186041 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.186020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tls-certs\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.186351 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.186334    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-4w59g\""
Sep 16 19:55:29.191189 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.191160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"]
Sep 16 19:55:29.195311 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.195287    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6qqz\" (UniqueName: \"kubernetes.io/projected/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kube-api-access-k6qqz\") pod \"tls-verification-test-kserve-598ffb6f9d-s956k\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.236378 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.236345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:29.284759 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.284724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1e1f41-333e-4a07-bf9d-993b7032502d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.284956 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.284768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94xnn\" (UniqueName: \"kubernetes.io/projected/fb1e1f41-333e-4a07-bf9d-993b7032502d-kube-api-access-94xnn\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.385317 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.385274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94xnn\" (UniqueName: \"kubernetes.io/projected/fb1e1f41-333e-4a07-bf9d-993b7032502d-kube-api-access-94xnn\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.385508 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.385410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1e1f41-333e-4a07-bf9d-993b7032502d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.387923 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.387902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1e1f41-333e-4a07-bf9d-993b7032502d-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.393966 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.393939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94xnn\" (UniqueName: \"kubernetes.io/projected/fb1e1f41-333e-4a07-bf9d-993b7032502d-kube-api-access-94xnn\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.510050 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.509964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:29.517607 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.517582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"]
Sep 16 19:55:29.523137 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:55:29.523102    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0096a6a_f7c6_4e7e_82ee_fb951217c2ec.slice/crio-4a4d0ae4dc1a02feca4d10de4d6c0b4046129f2295f8bdc5b9f2b0d6b933ee42 WatchSource:0}: Error finding container 4a4d0ae4dc1a02feca4d10de4d6c0b4046129f2295f8bdc5b9f2b0d6b933ee42: Status 404 returned error can't find the container with id 4a4d0ae4dc1a02feca4d10de4d6c0b4046129f2295f8bdc5b9f2b0d6b933ee42
Sep 16 19:55:29.794314 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:29.794283    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"]
Sep 16 19:55:29.796551 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:55:29.796511    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1e1f41_333e_4a07_bf9d_993b7032502d.slice/crio-38f4bbfad56b8e4cc607819d0502b525369855ccfedaba264d65f2f72596c452 WatchSource:0}: Error finding container 38f4bbfad56b8e4cc607819d0502b525369855ccfedaba264d65f2f72596c452: Status 404 returned error can't find the container with id 38f4bbfad56b8e4cc607819d0502b525369855ccfedaba264d65f2f72596c452
Sep 16 19:55:30.514176 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:30.514140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" event={"ID":"fb1e1f41-333e-4a07-bf9d-993b7032502d","Type":"ContainerStarted","Data":"a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae"}
Sep 16 19:55:30.514637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:30.514258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:55:30.514637 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:30.514291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" event={"ID":"fb1e1f41-333e-4a07-bf9d-993b7032502d","Type":"ContainerStarted","Data":"38f4bbfad56b8e4cc607819d0502b525369855ccfedaba264d65f2f72596c452"}
Sep 16 19:55:30.515840 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:30.515815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" event={"ID":"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec","Type":"ContainerStarted","Data":"54806e79818c7c7b460ff9b3da3febc17648e69d319a501149fa6c5ba956e20a"}
Sep 16 19:55:30.515992 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:30.515885    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" event={"ID":"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec","Type":"ContainerStarted","Data":"4a4d0ae4dc1a02feca4d10de4d6c0b4046129f2295f8bdc5b9f2b0d6b933ee42"}
Sep 16 19:55:30.532270 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:30.532224    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" podStartSLOduration=1.532211564 podStartE2EDuration="1.532211564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:30.530313106 +0000 UTC m=+4231.834330076" watchObservedRunningTime="2026-09-16 19:55:30.532211564 +0000 UTC m=+4231.836228525"
Sep 16 19:55:31.829928 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:31.829877    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:55:32.527714 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:32.527680    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerID="54806e79818c7c7b460ff9b3da3febc17648e69d319a501149fa6c5ba956e20a" exitCode=0
Sep 16 19:55:32.527929 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:32.527759    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" event={"ID":"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec","Type":"ContainerDied","Data":"54806e79818c7c7b460ff9b3da3febc17648e69d319a501149fa6c5ba956e20a"}
Sep 16 19:55:33.535106 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:33.535063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" event={"ID":"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec","Type":"ContainerStarted","Data":"c4a8760915c5c34ae8636b95ffc742a1655fe795051495a61c3a2a0b78804bed"}
Sep 16 19:55:33.560069 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:33.559999    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podStartSLOduration=3.144341718 podStartE2EDuration="5.559976316s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.415634598s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:33.556903985 +0000 UTC m=+4234.860920948" watchObservedRunningTime="2026-09-16 19:55:33.559976316 +0000 UTC m=+4234.863993284"
Sep 16 19:55:34.236827 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:34.236786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:39.237185 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:39.237151    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:55:39.239211 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:39.239175    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:55:41.830194 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:41.830144    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:55:49.237294 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:49.237247    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:55:51.829698 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:51.829645    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:55:59.237780 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:55:59.237737    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:56:01.523217 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:01.523187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:56:01.829783 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:01.829679    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:56:09.237628 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:09.237573    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:56:11.829449 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:11.829391    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:56:19.237578 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:19.237530    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:56:21.829163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:21.829116    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:56:29.237560 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:29.237517    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:56:31.829279 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:31.829230    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": dial tcp 10.132.0.91:8000: connect: connection refused"
Sep 16 19:56:39.237574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:39.237531    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:56:41.839321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:41.839285    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:56:41.847236 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:41.847209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:56:49.237545 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:49.237505    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:56:59.237718 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:56:59.237631    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:57:04.248374 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:04.248336    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"]
Sep 16 19:57:09.237687 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:09.237650    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:57:19.237435 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:19.237393    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:57:19.280234 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:19.280189    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main" containerID="cri-o://7f1224c04d62eb0438bbe6b9246230797960e0277d72ab703d375ce6475fc090" gracePeriod=45
Sep 16 19:57:20.175064 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.175023    2581 generic.go:356] "Generic (PLEG): container finished" podID="559bb1ff-11d6-4eac-adc0-87199e280821" containerID="7f1224c04d62eb0438bbe6b9246230797960e0277d72ab703d375ce6475fc090" exitCode=0
Sep 16 19:57:20.175254 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.175086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" event={"ID":"559bb1ff-11d6-4eac-adc0-87199e280821","Type":"ContainerDied","Data":"7f1224c04d62eb0438bbe6b9246230797960e0277d72ab703d375ce6475fc090"}
Sep 16 19:57:20.234139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.234112    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.239868    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/559bb1ff-11d6-4eac-adc0-87199e280821-cabundle-cert\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.239932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x78tv\" (UniqueName: \"kubernetes.io/projected/559bb1ff-11d6-4eac-adc0-87199e280821-kube-api-access-x78tv\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.239981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-model-cache\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.240019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-tmp-dir\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.240059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-kserve-provision-location\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.240115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-home\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.240323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bb1ff-11d6-4eac-adc0-87199e280821-tls-certs\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.240847 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.240371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-dshm\") pod \"559bb1ff-11d6-4eac-adc0-87199e280821\" (UID: \"559bb1ff-11d6-4eac-adc0-87199e280821\") "
Sep 16 19:57:20.242693 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.242653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-model-cache" (OuterVolumeSpecName: "model-cache") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.243000 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.242970    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/559bb1ff-11d6-4eac-adc0-87199e280821-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:57:20.248646 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.248389    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-dshm" (OuterVolumeSpecName: "dshm") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.248890 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.248829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559bb1ff-11d6-4eac-adc0-87199e280821-kube-api-access-x78tv" (OuterVolumeSpecName: "kube-api-access-x78tv") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "kube-api-access-x78tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:20.249670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.249638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/559bb1ff-11d6-4eac-adc0-87199e280821-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:20.250386 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.250364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-home" (OuterVolumeSpecName: "home") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.264987 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.264958    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.283572 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.283519    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "559bb1ff-11d6-4eac-adc0-87199e280821" (UID: "559bb1ff-11d6-4eac-adc0-87199e280821"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342448    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342478    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/559bb1ff-11d6-4eac-adc0-87199e280821-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342490    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x78tv\" (UniqueName: \"kubernetes.io/projected/559bb1ff-11d6-4eac-adc0-87199e280821-kube-api-access-x78tv\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342499    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342508    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342516    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342524    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/559bb1ff-11d6-4eac-adc0-87199e280821-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:20.342535 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:20.342532    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/559bb1ff-11d6-4eac-adc0-87199e280821-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:21.181474 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.181433    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v" event={"ID":"559bb1ff-11d6-4eac-adc0-87199e280821","Type":"ContainerDied","Data":"dea2fbbcf5155855881c86139c43d98f18c254c2e2d88932b27e9b4e7d460834"}
Sep 16 19:57:21.181674 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.181483    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"
Sep 16 19:57:21.181674 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.181538    2581 scope.go:117] "RemoveContainer" containerID="7f1224c04d62eb0438bbe6b9246230797960e0277d72ab703d375ce6475fc090"
Sep 16 19:57:21.190577 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.190554    2581 scope.go:117] "RemoveContainer" containerID="5d0b948b3ec54ce5c03f14dc5e61c7e902e84bc8195391e20036523e4621af82"
Sep 16 19:57:21.205159 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.205135    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"]
Sep 16 19:57:21.210463 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.210437    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-76cbfd4d99-kpp9v"]
Sep 16 19:57:21.221663 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:21.221638    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" path="/var/lib/kubelet/pods/559bb1ff-11d6-4eac-adc0-87199e280821/volumes"
Sep 16 19:57:29.247668 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:29.247631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:57:29.255669 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:29.255647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:57:34.574935 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:34.574893    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"]
Sep 16 19:57:34.581262 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:34.581227    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"]
Sep 16 19:57:41.522989 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:41.522949    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:49.595196 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:49.595158    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main" containerID="cri-o://a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae" gracePeriod=45
Sep 16 19:57:49.602620 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:49.602589    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" containerID="cri-o://c4a8760915c5c34ae8636b95ffc742a1655fe795051495a61c3a2a0b78804bed" gracePeriod=45
Sep 16 19:57:50.248314 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.248270    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Sep 16 19:57:50.329017 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.328984    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerID="c4a8760915c5c34ae8636b95ffc742a1655fe795051495a61c3a2a0b78804bed" exitCode=0
Sep 16 19:57:50.329190 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.329055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" event={"ID":"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec","Type":"ContainerDied","Data":"c4a8760915c5c34ae8636b95ffc742a1655fe795051495a61c3a2a0b78804bed"}
Sep 16 19:57:50.352292 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.352267    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:57:50.485550 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6qqz\" (UniqueName: \"kubernetes.io/projected/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kube-api-access-k6qqz\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.485719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485604    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-model-cache\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.485719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485622    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tmp-dir\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.485719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-dshm\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.485719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-cabundle-cert\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.485719 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485691    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tls-certs\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.486026 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kserve-provision-location\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.486026 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-home\") pod \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\" (UID: \"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec\") "
Sep 16 19:57:50.486026 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.485966    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:50.486409 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.486351    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:57:50.486635 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.486597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-home" (OuterVolumeSpecName: "home") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:50.488085 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.488029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kube-api-access-k6qqz" (OuterVolumeSpecName: "kube-api-access-k6qqz") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "kube-api-access-k6qqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:57:50.488085 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.488039    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-dshm" (OuterVolumeSpecName: "dshm") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:50.488424 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.488395    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:57:50.498040 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.498002    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:50.515476 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.515452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" (UID: "f0096a6a-f7c6-4e7e-82ee-fb951217c2ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:57:50.586604 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586580    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6qqz\" (UniqueName: \"kubernetes.io/projected/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kube-api-access-k6qqz\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586605    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586622    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586633    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586645    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586656    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586669    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:50.586712 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:50.586683    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:57:51.335720 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.335689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"
Sep 16 19:57:51.336173 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.335719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k" event={"ID":"f0096a6a-f7c6-4e7e-82ee-fb951217c2ec","Type":"ContainerDied","Data":"4a4d0ae4dc1a02feca4d10de4d6c0b4046129f2295f8bdc5b9f2b0d6b933ee42"}
Sep 16 19:57:51.336173 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.335779    2581 scope.go:117] "RemoveContainer" containerID="c4a8760915c5c34ae8636b95ffc742a1655fe795051495a61c3a2a0b78804bed"
Sep 16 19:57:51.344886 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.344847    2581 scope.go:117] "RemoveContainer" containerID="54806e79818c7c7b460ff9b3da3febc17648e69d319a501149fa6c5ba956e20a"
Sep 16 19:57:51.357688 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.357664    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"]
Sep 16 19:57:51.363342 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.363317    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-598ffb6f9d-s956k"]
Sep 16 19:57:51.523665 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:51.523585    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:53.221670 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:57:53.221634    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" path="/var/lib/kubelet/pods/f0096a6a-f7c6-4e7e-82ee-fb951217c2ec/volumes"
Sep 16 19:58:01.523506 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:01.523464    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:58:01.523921 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:01.523565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:58:05.801738 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.801700    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"]
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802053    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802068    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802079    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="storage-initializer"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802085    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="storage-initializer"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802092    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802097    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="storage-initializer"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="storage-initializer"
Sep 16 19:58:05.802154 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802158    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="559bb1ff-11d6-4eac-adc0-87199e280821" containerName="main"
Sep 16 19:58:05.802482 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.802168    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0096a6a-f7c6-4e7e-82ee-fb951217c2ec" containerName="main"
Sep 16 19:58:05.806463 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.806434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.809446 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.809420    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:58:05.809560 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.809443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:05.809560 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.809478    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:58:05.809560 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.809512    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:58:05.809762 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.809566    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-wm6n4\""
Sep 16 19:58:05.809762 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.809654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:05.820288 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.820264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"]
Sep 16 19:58:05.905444 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905609 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7wt\" (UniqueName: \"kubernetes.io/projected/8229a592-98b1-4753-9c4b-2b117125f881-kube-api-access-wn7wt\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905609 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229a592-98b1-4753-9c4b-2b117125f881-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905609 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905742 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-home\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905742 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905742 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8229a592-98b1-4753-9c4b-2b117125f881-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:05.905742 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:05.905707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.006930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.006889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-home\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.006930 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.006934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.006952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8229a592-98b1-4753-9c4b-2b117125f881-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.006977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7wt\" (UniqueName: \"kubernetes.io/projected/8229a592-98b1-4753-9c4b-2b117125f881-kube-api-access-wn7wt\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229a592-98b1-4753-9c4b-2b117125f881-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007193 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007186    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-home\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007505 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007606 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007705 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.007844 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.007723    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8229a592-98b1-4753-9c4b-2b117125f881-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.009357 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.009335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.009801 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.009780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229a592-98b1-4753-9c4b-2b117125f881-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.017280 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.017257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7wt\" (UniqueName: \"kubernetes.io/projected/8229a592-98b1-4753-9c4b-2b117125f881-kube-api-access-wn7wt\") pod \"router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.116582 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.116497    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:06.396576 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.396545    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"]
Sep 16 19:58:06.397419 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:58:06.397390    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8229a592_98b1_4753_9c4b_2b117125f881.slice/crio-764349cd8e406eba2100334296473915a889b40709679e3138a276c87da0909a WatchSource:0}: Error finding container 764349cd8e406eba2100334296473915a889b40709679e3138a276c87da0909a: Status 404 returned error can't find the container with id 764349cd8e406eba2100334296473915a889b40709679e3138a276c87da0909a
Sep 16 19:58:06.410792 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:06.410764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" event={"ID":"8229a592-98b1-4753-9c4b-2b117125f881","Type":"ContainerStarted","Data":"764349cd8e406eba2100334296473915a889b40709679e3138a276c87da0909a"}
Sep 16 19:58:07.417139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:07.417099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" event={"ID":"8229a592-98b1-4753-9c4b-2b117125f881","Type":"ContainerStarted","Data":"42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f"}
Sep 16 19:58:09.430610 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:09.430582    2581 generic.go:356] "Generic (PLEG): container finished" podID="8229a592-98b1-4753-9c4b-2b117125f881" containerID="42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f" exitCode=0
Sep 16 19:58:09.430994 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:09.430656    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" event={"ID":"8229a592-98b1-4753-9c4b-2b117125f881","Type":"ContainerDied","Data":"42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f"}
Sep 16 19:58:10.437693 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:10.437650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" event={"ID":"8229a592-98b1-4753-9c4b-2b117125f881","Type":"ContainerStarted","Data":"3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2"}
Sep 16 19:58:10.462244 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:10.462191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podStartSLOduration=2.979730408 podStartE2EDuration="5.462176702s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.482446294s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:10.46064114 +0000 UTC m=+4391.764658109" watchObservedRunningTime="2026-09-16 19:58:10.462176702 +0000 UTC m=+4391.766193669"
Sep 16 19:58:11.523324 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:11.523288    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:58:16.117557 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:16.117515    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:16.118077 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:16.117596    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 19:58:16.119119 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:16.119089    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:58:19.848760 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:19.848730    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:58:19.925321 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:19.925279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94xnn\" (UniqueName: \"kubernetes.io/projected/fb1e1f41-333e-4a07-bf9d-993b7032502d-kube-api-access-94xnn\") pod \"fb1e1f41-333e-4a07-bf9d-993b7032502d\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") "
Sep 16 19:58:19.925520 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:19.925336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1e1f41-333e-4a07-bf9d-993b7032502d-tls-certs\") pod \"fb1e1f41-333e-4a07-bf9d-993b7032502d\" (UID: \"fb1e1f41-333e-4a07-bf9d-993b7032502d\") "
Sep 16 19:58:19.927326 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:19.927300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1e1f41-333e-4a07-bf9d-993b7032502d-kube-api-access-94xnn" (OuterVolumeSpecName: "kube-api-access-94xnn") pod "fb1e1f41-333e-4a07-bf9d-993b7032502d" (UID: "fb1e1f41-333e-4a07-bf9d-993b7032502d"). InnerVolumeSpecName "kube-api-access-94xnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:58:19.927466 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:19.927401    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1e1f41-333e-4a07-bf9d-993b7032502d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb1e1f41-333e-4a07-bf9d-993b7032502d" (UID: "fb1e1f41-333e-4a07-bf9d-993b7032502d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:58:20.026585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.026536    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94xnn\" (UniqueName: \"kubernetes.io/projected/fb1e1f41-333e-4a07-bf9d-993b7032502d-kube-api-access-94xnn\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:58:20.026585 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.026567    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb1e1f41-333e-4a07-bf9d-993b7032502d-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 19:58:20.490583 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.490547    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerID="a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae" exitCode=0
Sep 16 19:58:20.490752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.490589    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"
Sep 16 19:58:20.490752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.490632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" event={"ID":"fb1e1f41-333e-4a07-bf9d-993b7032502d","Type":"ContainerDied","Data":"a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae"}
Sep 16 19:58:20.490752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.490678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr" event={"ID":"fb1e1f41-333e-4a07-bf9d-993b7032502d","Type":"ContainerDied","Data":"38f4bbfad56b8e4cc607819d0502b525369855ccfedaba264d65f2f72596c452"}
Sep 16 19:58:20.490752 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.490700    2581 scope.go:117] "RemoveContainer" containerID="a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae"
Sep 16 19:58:20.499721 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.499705    2581 scope.go:117] "RemoveContainer" containerID="a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae"
Sep 16 19:58:20.500067 ip-10-0-133-194 kubenswrapper[2581]: E0916 19:58:20.500046    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae\": container with ID starting with a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae not found: ID does not exist" containerID="a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae"
Sep 16 19:58:20.500153 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.500087    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae"} err="failed to get container status \"a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae\": rpc error: code = NotFound desc = could not find container \"a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae\": container with ID starting with a490ccfc1bd0462ee2b409260d13f51e3daa05b146d4dc724f9224a2ba2c29ae not found: ID does not exist"
Sep 16 19:58:20.512784 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.512761    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"]
Sep 16 19:58:20.516837 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:20.516813    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-49jsr"]
Sep 16 19:58:21.223053 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:21.223012    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" path="/var/lib/kubelet/pods/fb1e1f41-333e-4a07-bf9d-993b7032502d/volumes"
Sep 16 19:58:26.117548 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:26.117504    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:58:36.117204 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:36.117156    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:58:43.620139 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.620103    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"]
Sep 16 19:58:43.620580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.620416    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main"
Sep 16 19:58:43.620580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.620428    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main"
Sep 16 19:58:43.620580 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.620492    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb1e1f41-333e-4a07-bf9d-993b7032502d" containerName="main"
Sep 16 19:58:43.623964 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.623941    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.626919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.626741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 19:58:43.626919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.626761    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-42p9m\""
Sep 16 19:58:43.626919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.626760    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:58:43.626919 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.626891    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:58:43.627199 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.626900    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:58:43.627199 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.627106    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:58:43.637932 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.637910    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"]
Sep 16 19:58:43.736369 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d377194a-618a-4b13-a07f-8ad970ed3897-cabundle-cert\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-home\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-model-cache\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-dshm\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-kserve-provision-location\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-tmp-dir\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736556 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv757\" (UniqueName: \"kubernetes.io/projected/d377194a-618a-4b13-a07f-8ad970ed3897-kube-api-access-fv757\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.736824 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.736610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377194a-618a-4b13-a07f-8ad970ed3897-tls-certs\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.837901 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.837844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d377194a-618a-4b13-a07f-8ad970ed3897-cabundle-cert\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.837901 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.837908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-home\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.837926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-model-cache\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.837947    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-dshm\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.837962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-kserve-provision-location\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.837978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-tmp-dir\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv757\" (UniqueName: \"kubernetes.io/projected/d377194a-618a-4b13-a07f-8ad970ed3897-kube-api-access-fv757\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838166 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377194a-618a-4b13-a07f-8ad970ed3897-tls-certs\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-home\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838650 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-tmp-dir\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838718    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d377194a-618a-4b13-a07f-8ad970ed3897-cabundle-cert\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838877 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-kserve-provision-location\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.838940 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.838839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-model-cache\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.840574 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.840542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-dshm\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.840926 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.840904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377194a-618a-4b13-a07f-8ad970ed3897-tls-certs\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.845787 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.845762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv757\" (UniqueName: \"kubernetes.io/projected/d377194a-618a-4b13-a07f-8ad970ed3897-kube-api-access-fv757\") pod \"prestop-hook-test-kserve-fff447946-txd7h\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:43.911195 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.911153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"]
Sep 16 19:58:43.919757 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.919729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:43.923182 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.923153    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-jqhjf\""
Sep 16 19:58:43.923634 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.923609    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"]
Sep 16 19:58:43.935309 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:43.935245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:44.040546 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.040515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmpqr\" (UniqueName: \"kubernetes.io/projected/a46788f9-0ccc-4309-9dc0-dc4bee9be459-kube-api-access-nmpqr\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.040723 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.040676    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46788f9-0ccc-4309-9dc0-dc4bee9be459-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.141601 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.141566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46788f9-0ccc-4309-9dc0-dc4bee9be459-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.141792 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.141637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmpqr\" (UniqueName: \"kubernetes.io/projected/a46788f9-0ccc-4309-9dc0-dc4bee9be459-kube-api-access-nmpqr\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.144171 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.144141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46788f9-0ccc-4309-9dc0-dc4bee9be459-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.151942 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.151917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmpqr\" (UniqueName: \"kubernetes.io/projected/a46788f9-0ccc-4309-9dc0-dc4bee9be459-kube-api-access-nmpqr\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.223418 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.223390    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"]
Sep 16 19:58:44.224590 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:58:44.224566    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd377194a_618a_4b13_a07f_8ad970ed3897.slice/crio-cb8959b737c108a70242423304cbe47f752db6da5ee53031721feca8075072e3 WatchSource:0}: Error finding container cb8959b737c108a70242423304cbe47f752db6da5ee53031721feca8075072e3: Status 404 returned error can't find the container with id cb8959b737c108a70242423304cbe47f752db6da5ee53031721feca8075072e3
Sep 16 19:58:44.232651 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.232633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:44.538366 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.538328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"]
Sep 16 19:58:44.542536 ip-10-0-133-194 kubenswrapper[2581]: W0916 19:58:44.542501    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda46788f9_0ccc_4309_9dc0_dc4bee9be459.slice/crio-4a0ecaa8daf8b96075d869cb184a849b5ab3bac81d80e4810da208c6b195cb0d WatchSource:0}: Error finding container 4a0ecaa8daf8b96075d869cb184a849b5ab3bac81d80e4810da208c6b195cb0d: Status 404 returned error can't find the container with id 4a0ecaa8daf8b96075d869cb184a849b5ab3bac81d80e4810da208c6b195cb0d
Sep 16 19:58:44.616790 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.616753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" event={"ID":"d377194a-618a-4b13-a07f-8ad970ed3897","Type":"ContainerStarted","Data":"50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21"}
Sep 16 19:58:44.616977 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.616818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" event={"ID":"d377194a-618a-4b13-a07f-8ad970ed3897","Type":"ContainerStarted","Data":"cb8959b737c108a70242423304cbe47f752db6da5ee53031721feca8075072e3"}
Sep 16 19:58:44.618073 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:44.618038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" event={"ID":"a46788f9-0ccc-4309-9dc0-dc4bee9be459","Type":"ContainerStarted","Data":"4a0ecaa8daf8b96075d869cb184a849b5ab3bac81d80e4810da208c6b195cb0d"}
Sep 16 19:58:45.625043 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:45.624997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" event={"ID":"a46788f9-0ccc-4309-9dc0-dc4bee9be459","Type":"ContainerStarted","Data":"d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23"}
Sep 16 19:58:45.625520 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:45.625100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:58:46.117267 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:46.117213    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:58:47.636526 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:47.636491    2581 generic.go:356] "Generic (PLEG): container finished" podID="d377194a-618a-4b13-a07f-8ad970ed3897" containerID="50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21" exitCode=0
Sep 16 19:58:47.636990 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:47.636554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" event={"ID":"d377194a-618a-4b13-a07f-8ad970ed3897","Type":"ContainerDied","Data":"50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21"}
Sep 16 19:58:47.666128 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:47.666045    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podStartSLOduration=4.666022595 podStartE2EDuration="4.666022595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:45.646545719 +0000 UTC m=+4426.950562688" watchObservedRunningTime="2026-09-16 19:58:47.666022595 +0000 UTC m=+4428.970039563"
Sep 16 19:58:48.644435 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:48.644385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" event={"ID":"d377194a-618a-4b13-a07f-8ad970ed3897","Type":"ContainerStarted","Data":"d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40"}
Sep 16 19:58:48.936187 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:48.936152    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:53.935741 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:53.935703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 19:58:53.937489 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:53.937455    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:58:56.117196 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:56.117142    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:58:59.909800 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:58:59.909764    2581 scope.go:117] "RemoveContainer" containerID="090acf943cf86c808d888c92227201149a86489fc590229eff7116e436dfdfb2"
Sep 16 19:59:03.935692 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:03.935647    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:59:06.117512 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:06.117462    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:59:13.935970 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:13.935921    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:59:16.117701 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:16.117661    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:59:16.632108 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:16.632065    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 19:59:16.651122 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:16.651073    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podStartSLOduration=30.887710529 podStartE2EDuration="33.65105922s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.763348691s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:48.683707964 +0000 UTC m=+4429.987724932" watchObservedRunningTime="2026-09-16 19:59:16.65105922 +0000 UTC m=+4457.955076187"
Sep 16 19:59:23.936742 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:23.936695    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:59:26.116990 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:26.116936    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:59:33.936242 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:33.936184    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:59:36.117163 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:36.117111    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:59:43.936610 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:43.936550    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:59:46.117181 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:46.117135    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:59:53.935796 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:53.935696    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 19:59:56.117667 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:56.117618    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" probeResult="failure" output="Get \"https://10.132.0.94:8000/health\": dial tcp 10.132.0.94:8000: connect: connection refused"
Sep 16 19:59:59.639795 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:59.639761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:59:59.643439 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:59.643412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 19:59:59.651658 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:59.651635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 19:59:59.655009 ip-10-0-133-194 kubenswrapper[2581]: I0916 19:59:59.654984    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lwq4x_3d97f496-2126-40cb-b3dd-763207a65c4b/ovn-acl-logging/0.log"
Sep 16 20:00:03.936628 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:03.936573    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 20:00:06.127177 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:06.127141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 20:00:06.135185 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:06.135153    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 20:00:13.936516 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:13.936467    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 20:00:19.661789 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:19.661754    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"]
Sep 16 20:00:23.936659 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:23.936610    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 20:00:33.936216 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:33.936174    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 20:00:34.685490 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:34.685439    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main" containerID="cri-o://3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2" gracePeriod=45
Sep 16 20:00:35.629348 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.629319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 20:00:35.730911 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.730801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-dshm\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.730911 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.730888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-home\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731138 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.730919    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-kserve-provision-location\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731138 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.730954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-model-cache\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731138 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.730989    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8229a592-98b1-4753-9c4b-2b117125f881-cabundle-cert\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731138 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.731014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-tmp-dir\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731138 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.731043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229a592-98b1-4753-9c4b-2b117125f881-tls-certs\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731138 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.731100    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn7wt\" (UniqueName: \"kubernetes.io/projected/8229a592-98b1-4753-9c4b-2b117125f881-kube-api-access-wn7wt\") pod \"8229a592-98b1-4753-9c4b-2b117125f881\" (UID: \"8229a592-98b1-4753-9c4b-2b117125f881\") "
Sep 16 20:00:35.731627 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.731565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-home" (OuterVolumeSpecName: "home") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.731965 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.731808    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-model-cache" (OuterVolumeSpecName: "model-cache") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.731965 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.731931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8229a592-98b1-4753-9c4b-2b117125f881-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:00:35.733350 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.733317    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-dshm" (OuterVolumeSpecName: "dshm") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.733768 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.733745    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8229a592-98b1-4753-9c4b-2b117125f881-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:00:35.733881 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.733752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8229a592-98b1-4753-9c4b-2b117125f881-kube-api-access-wn7wt" (OuterVolumeSpecName: "kube-api-access-wn7wt") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "kube-api-access-wn7wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:00:35.747234 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.747195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.768906 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.768837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8229a592-98b1-4753-9c4b-2b117125f881" (UID: "8229a592-98b1-4753-9c4b-2b117125f881"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:00:35.832544 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832511    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8229a592-98b1-4753-9c4b-2b117125f881-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832544 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832541    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7wt\" (UniqueName: \"kubernetes.io/projected/8229a592-98b1-4753-9c4b-2b117125f881-kube-api-access-wn7wt\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832707 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832557    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832707 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832570    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832707 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832581    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832707 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832594    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832707 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832606    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8229a592-98b1-4753-9c4b-2b117125f881-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:35.832707 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:35.832616    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8229a592-98b1-4753-9c4b-2b117125f881-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:00:36.261479 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.261441    2581 generic.go:356] "Generic (PLEG): container finished" podID="8229a592-98b1-4753-9c4b-2b117125f881" containerID="3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2" exitCode=0
Sep 16 20:00:36.261644 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.261539    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"
Sep 16 20:00:36.261644 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.261534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" event={"ID":"8229a592-98b1-4753-9c4b-2b117125f881","Type":"ContainerDied","Data":"3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2"}
Sep 16 20:00:36.261725 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.261704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf" event={"ID":"8229a592-98b1-4753-9c4b-2b117125f881","Type":"ContainerDied","Data":"764349cd8e406eba2100334296473915a889b40709679e3138a276c87da0909a"}
Sep 16 20:00:36.261761 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.261728    2581 scope.go:117] "RemoveContainer" containerID="3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2"
Sep 16 20:00:36.270794 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.270769    2581 scope.go:117] "RemoveContainer" containerID="42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f"
Sep 16 20:00:36.278805 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.278779    2581 scope.go:117] "RemoveContainer" containerID="3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2"
Sep 16 20:00:36.279102 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:00:36.279075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2\": container with ID starting with 3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2 not found: ID does not exist" containerID="3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2"
Sep 16 20:00:36.279195 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.279114    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2"} err="failed to get container status \"3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2\": rpc error: code = NotFound desc = could not find container \"3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2\": container with ID starting with 3df5a877548e256459647ad078c250ff20c0c58667889e8322ec694c68710df2 not found: ID does not exist"
Sep 16 20:00:36.279195 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.279139    2581 scope.go:117] "RemoveContainer" containerID="42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f"
Sep 16 20:00:36.279379 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:00:36.279359    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f\": container with ID starting with 42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f not found: ID does not exist" containerID="42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f"
Sep 16 20:00:36.279442 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.279384    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f"} err="failed to get container status \"42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f\": rpc error: code = NotFound desc = could not find container \"42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f\": container with ID starting with 42c1784c5d17791b0ee4c77152b7872270b09f76d4fa46e1386983e0d64f918f not found: ID does not exist"
Sep 16 20:00:36.293413 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.293389    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"]
Sep 16 20:00:36.301901 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:36.301878    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-849f57dbfc-lm7lf"]
Sep 16 20:00:37.221720 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:37.221687    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8229a592-98b1-4753-9c4b-2b117125f881" path="/var/lib/kubelet/pods/8229a592-98b1-4753-9c4b-2b117125f881/volumes"
Sep 16 20:00:43.945636 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:43.945599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 20:00:43.953136 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:43.953108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 20:00:45.789301 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:00:45.789267    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"]
Sep 16 20:01:00.811769 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:00.811726    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" containerID="cri-o://d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40" gracePeriod=45
Sep 16 20:01:00.947081 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:00.947022    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": dial tcp 10.132.0.95:8000: connect: connection refused"
Sep 16 20:01:01.556222 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.556192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 20:01:01.627156 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-dshm\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627156 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-model-cache\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627391 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377194a-618a-4b13-a07f-8ad970ed3897-tls-certs\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627391 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv757\" (UniqueName: \"kubernetes.io/projected/d377194a-618a-4b13-a07f-8ad970ed3897-kube-api-access-fv757\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627510 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d377194a-618a-4b13-a07f-8ad970ed3897-cabundle-cert\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627510 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-model-cache" (OuterVolumeSpecName: "model-cache") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:01.627510 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-kserve-provision-location\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627674 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-home\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627674 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-tmp-dir\") pod \"d377194a-618a-4b13-a07f-8ad970ed3897\" (UID: \"d377194a-618a-4b13-a07f-8ad970ed3897\") "
Sep 16 20:01:01.627822 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627795    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d377194a-618a-4b13-a07f-8ad970ed3897-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:01:01.628016 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.627995    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.628092 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.628023    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d377194a-618a-4b13-a07f-8ad970ed3897-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.628308 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.628253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-home" (OuterVolumeSpecName: "home") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:01.629961 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.629923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d377194a-618a-4b13-a07f-8ad970ed3897-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:01.630055 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.630011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-dshm" (OuterVolumeSpecName: "dshm") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:01.630165 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.630049    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d377194a-618a-4b13-a07f-8ad970ed3897-kube-api-access-fv757" (OuterVolumeSpecName: "kube-api-access-fv757") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "kube-api-access-fv757". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:01.639698 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.639668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:01.650152 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.650120    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d377194a-618a-4b13-a07f-8ad970ed3897" (UID: "d377194a-618a-4b13-a07f-8ad970ed3897"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:01:01.729240 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.729199    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.729240 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.729237    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.729240 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.729249    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.729496 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.729260    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d377194a-618a-4b13-a07f-8ad970ed3897-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.729496 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.729271    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv757\" (UniqueName: \"kubernetes.io/projected/d377194a-618a-4b13-a07f-8ad970ed3897-kube-api-access-fv757\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:01.729496 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:01.729283    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d377194a-618a-4b13-a07f-8ad970ed3897-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:02.399270 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.399230    2581 generic.go:356] "Generic (PLEG): container finished" podID="d377194a-618a-4b13-a07f-8ad970ed3897" containerID="d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40" exitCode=0
Sep 16 20:01:02.399703 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.399281    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"
Sep 16 20:01:02.399703 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.399304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" event={"ID":"d377194a-618a-4b13-a07f-8ad970ed3897","Type":"ContainerDied","Data":"d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40"}
Sep 16 20:01:02.399703 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.399355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h" event={"ID":"d377194a-618a-4b13-a07f-8ad970ed3897","Type":"ContainerDied","Data":"cb8959b737c108a70242423304cbe47f752db6da5ee53031721feca8075072e3"}
Sep 16 20:01:02.399703 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.399377    2581 scope.go:117] "RemoveContainer" containerID="d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40"
Sep 16 20:01:02.408554 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.408305    2581 scope.go:117] "RemoveContainer" containerID="50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21"
Sep 16 20:01:02.416178 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.416156    2581 scope.go:117] "RemoveContainer" containerID="d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40"
Sep 16 20:01:02.416457 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:01:02.416438    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40\": container with ID starting with d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40 not found: ID does not exist" containerID="d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40"
Sep 16 20:01:02.416518 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.416466    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40"} err="failed to get container status \"d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40\": rpc error: code = NotFound desc = could not find container \"d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40\": container with ID starting with d6d6e05294be57ceec1728f86d2c05b97c5ffec48141ca037a97eca9798e8f40 not found: ID does not exist"
Sep 16 20:01:02.416518 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.416484    2581 scope.go:117] "RemoveContainer" containerID="50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21"
Sep 16 20:01:02.416740 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:01:02.416721    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21\": container with ID starting with 50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21 not found: ID does not exist" containerID="50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21"
Sep 16 20:01:02.416795 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.416749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21"} err="failed to get container status \"50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21\": rpc error: code = NotFound desc = could not find container \"50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21\": container with ID starting with 50a854df2671b3274e52956e85a5df3b1882f507b986034bb246eff90d8a2c21 not found: ID does not exist"
Sep 16 20:01:02.425674 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.425652    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"]
Sep 16 20:01:02.432047 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:02.432025    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-fff447946-txd7h"]
Sep 16 20:01:03.222011 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:03.221975    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" path="/var/lib/kubelet/pods/d377194a-618a-4b13-a07f-8ad970ed3897/volumes"
Sep 16 20:01:04.157875 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:04.157832    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"]
Sep 16 20:01:06.632516 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:06.632473    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:16.632032 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:16.631991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:19.179849 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:19.179798    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main" containerID="cri-o://d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23" gracePeriod=45
Sep 16 20:01:20.421729 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.421687    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"]
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422362    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422401    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422442    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="storage-initializer"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422452    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="storage-initializer"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422478    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="storage-initializer"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422487    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="storage-initializer"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d377194a-618a-4b13-a07f-8ad970ed3897" containerName="main"
Sep 16 20:01:20.424492 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.422647    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8229a592-98b1-4753-9c4b-2b117125f881" containerName="main"
Sep 16 20:01:20.427845 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.427821    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.430789 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.430762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 20:01:20.430910 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.430771    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 20:01:20.430910 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.430832    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 20:01:20.431015 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.430908    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-v8m9k\""
Sep 16 20:01:20.431015 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.430911    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 20:01:20.431125 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.431111    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 20:01:20.437688 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.437667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"]
Sep 16 20:01:20.468473 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468473 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468635 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9c7849d-ccfb-4be8-990e-ad015ff6554b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468635 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468635 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468635 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kube-api-access-b6tbf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468635 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468635    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.468842 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.468684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569460 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569633 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569633 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kube-api-access-b6tbf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569633 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569633 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569633 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569962 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569962 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9c7849d-ccfb-4be8-990e-ad015ff6554b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569962 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.569962 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.569934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.570182 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.570098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.570182 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.570128    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.570436 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.570415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9c7849d-ccfb-4be8-990e-ad015ff6554b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.571936 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.571915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.572145 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.572127    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.580363 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.580334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kube-api-access-b6tbf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:20.739290 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:20.739197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:21.023645 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:21.023607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"]
Sep 16 20:01:21.024744 ip-10-0-133-194 kubenswrapper[2581]: W0916 20:01:21.024706    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c7849d_ccfb_4be8_990e_ad015ff6554b.slice/crio-5c639c603e9bed44ac558f42811e1118fe262fb69169fc9129da1586103add2d WatchSource:0}: Error finding container 5c639c603e9bed44ac558f42811e1118fe262fb69169fc9129da1586103add2d: Status 404 returned error can't find the container with id 5c639c603e9bed44ac558f42811e1118fe262fb69169fc9129da1586103add2d
Sep 16 20:01:21.503737 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:21.503694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" event={"ID":"a9c7849d-ccfb-4be8-990e-ad015ff6554b","Type":"ContainerStarted","Data":"3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015"}
Sep 16 20:01:21.504134 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:21.503756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" event={"ID":"a9c7849d-ccfb-4be8-990e-ad015ff6554b","Type":"ContainerStarted","Data":"5c639c603e9bed44ac558f42811e1118fe262fb69169fc9129da1586103add2d"}
Sep 16 20:01:23.514330 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:23.514294    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerID="3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015" exitCode=0
Sep 16 20:01:23.514692 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:23.514367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" event={"ID":"a9c7849d-ccfb-4be8-990e-ad015ff6554b","Type":"ContainerDied","Data":"3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015"}
Sep 16 20:01:24.522303 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:24.522259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" event={"ID":"a9c7849d-ccfb-4be8-990e-ad015ff6554b","Type":"ContainerStarted","Data":"5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae"}
Sep 16 20:01:24.546670 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:24.546609    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podStartSLOduration=2.210267973 podStartE2EDuration="4.546593232s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.336325259s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:01:24.54518979 +0000 UTC m=+4585.849206761" watchObservedRunningTime="2026-09-16 20:01:24.546593232 +0000 UTC m=+4585.850610255"
Sep 16 20:01:24.740232 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:24.740188    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:26.632288 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:26.632245    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:26.632745 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:26.632347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 20:01:30.739728 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:30.739695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:01:30.741142 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:30.741116    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:01:36.632659 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:36.632621    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:40.740095 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:40.740045    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:01:46.632260 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:46.632215    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:49.431716 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.431689    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 20:01:49.518602 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.518504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmpqr\" (UniqueName: \"kubernetes.io/projected/a46788f9-0ccc-4309-9dc0-dc4bee9be459-kube-api-access-nmpqr\") pod \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") "
Sep 16 20:01:49.518602 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.518568    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46788f9-0ccc-4309-9dc0-dc4bee9be459-tls-certs\") pod \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\" (UID: \"a46788f9-0ccc-4309-9dc0-dc4bee9be459\") "
Sep 16 20:01:49.520689 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.520653    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a46788f9-0ccc-4309-9dc0-dc4bee9be459-kube-api-access-nmpqr" (OuterVolumeSpecName: "kube-api-access-nmpqr") pod "a46788f9-0ccc-4309-9dc0-dc4bee9be459" (UID: "a46788f9-0ccc-4309-9dc0-dc4bee9be459"). InnerVolumeSpecName "kube-api-access-nmpqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:49.520805 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.520685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a46788f9-0ccc-4309-9dc0-dc4bee9be459-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a46788f9-0ccc-4309-9dc0-dc4bee9be459" (UID: "a46788f9-0ccc-4309-9dc0-dc4bee9be459"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:49.619955 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.619917    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmpqr\" (UniqueName: \"kubernetes.io/projected/a46788f9-0ccc-4309-9dc0-dc4bee9be459-kube-api-access-nmpqr\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:49.619955 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.619949    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a46788f9-0ccc-4309-9dc0-dc4bee9be459-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:01:49.652356 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.652320    2581 generic.go:356] "Generic (PLEG): container finished" podID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerID="d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23" exitCode=0
Sep 16 20:01:49.652502 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.652384    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"
Sep 16 20:01:49.652502 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.652405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" event={"ID":"a46788f9-0ccc-4309-9dc0-dc4bee9be459","Type":"ContainerDied","Data":"d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23"}
Sep 16 20:01:49.652502 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.652458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8" event={"ID":"a46788f9-0ccc-4309-9dc0-dc4bee9be459","Type":"ContainerDied","Data":"4a0ecaa8daf8b96075d869cb184a849b5ab3bac81d80e4810da208c6b195cb0d"}
Sep 16 20:01:49.652502 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.652480    2581 scope.go:117] "RemoveContainer" containerID="d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23"
Sep 16 20:01:49.660974 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.660947    2581 scope.go:117] "RemoveContainer" containerID="d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23"
Sep 16 20:01:49.661247 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:01:49.661225    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23\": container with ID starting with d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23 not found: ID does not exist" containerID="d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23"
Sep 16 20:01:49.661329 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.661255    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23"} err="failed to get container status \"d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23\": rpc error: code = NotFound desc = could not find container \"d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23\": container with ID starting with d066a0a1d3e9baa7902920e8c0a6940c4c29bdedafe8abc88431dcb3cdfefb23 not found: ID does not exist"
Sep 16 20:01:49.678300 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.678270    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"]
Sep 16 20:01:49.683910 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:49.683885    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-cjlx8"]
Sep 16 20:01:50.739965 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:50.739926    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:01:51.221327 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:01:51.221291    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" path="/var/lib/kubelet/pods/a46788f9-0ccc-4309-9dc0-dc4bee9be459/volumes"
Sep 16 20:02:00.739664 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:00.739619    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:02:10.740068 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:10.740021    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:02:20.739905 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:20.739819    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:02:30.740414 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:30.740369    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:02:40.739849 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:40.739798    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:02:48.591240 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.591206    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"]
Sep 16 20:02:48.591729 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.591543    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main"
Sep 16 20:02:48.591729 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.591554    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main"
Sep 16 20:02:48.591729 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.591633    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a46788f9-0ccc-4309-9dc0-dc4bee9be459" containerName="main"
Sep 16 20:02:48.594915 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.594894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.598495 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.598471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:02:48.598495 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.598488    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:02:48.598695 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.598549    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:02:48.598695 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.598488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-tmh5s\""
Sep 16 20:02:48.605783 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.605760    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"]
Sep 16 20:02:48.733605 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.733568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.733605 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.733602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.733831 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.733620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.733831 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.733645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chp4z\" (UniqueName: \"kubernetes.io/projected/4cfcfafe-2f29-4161-b095-1f099f33db36-kube-api-access-chp4z\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.733831 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.733691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.733831 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.733714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.834796 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.834756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.834796 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.834798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835078 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.834826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835078 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.834891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chp4z\" (UniqueName: \"kubernetes.io/projected/4cfcfafe-2f29-4161-b095-1f099f33db36-kube-api-access-chp4z\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835078 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.834927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835078 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.834967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835317 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.835281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835317 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.835293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.835518 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.835358    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.837282 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.837258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.837659 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.837638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.843371 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.843320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chp4z\" (UniqueName: \"kubernetes.io/projected/4cfcfafe-2f29-4161-b095-1f099f33db36-kube-api-access-chp4z\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-x7ffg\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:48.905938 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:48.905894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:49.182616 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:49.182581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"]
Sep 16 20:02:49.183466 ip-10-0-133-194 kubenswrapper[2581]: W0916 20:02:49.183435    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfcfafe_2f29_4161_b095_1f099f33db36.slice/crio-122c67273b25c2281320a7f68665161096697b44d2ea4211b0884e143c47d973 WatchSource:0}: Error finding container 122c67273b25c2281320a7f68665161096697b44d2ea4211b0884e143c47d973: Status 404 returned error can't find the container with id 122c67273b25c2281320a7f68665161096697b44d2ea4211b0884e143c47d973
Sep 16 20:02:49.965793 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:49.965750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" event={"ID":"4cfcfafe-2f29-4161-b095-1f099f33db36","Type":"ContainerStarted","Data":"c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"}
Sep 16 20:02:49.966192 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:49.965808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" event={"ID":"4cfcfafe-2f29-4161-b095-1f099f33db36","Type":"ContainerStarted","Data":"122c67273b25c2281320a7f68665161096697b44d2ea4211b0884e143c47d973"}
Sep 16 20:02:49.985278 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:49.985222    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" podStartSLOduration=1.985206851 podStartE2EDuration="1.985206851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:02:49.984347253 +0000 UTC m=+4671.288364232" watchObservedRunningTime="2026-09-16 20:02:49.985206851 +0000 UTC m=+4671.289223817"
Sep 16 20:02:50.740384 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:50.740339    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:02:50.906940 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:50.906900    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:58.906080 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:58.905993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:58.919207 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:58.919172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:59.040729 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:59.040690    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:02:59.339714 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:02:59.339624    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"]
Sep 16 20:03:00.121247 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:00.121214    2581 secret.go:189] Couldn't get secret e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 16 20:03:00.121659 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:00.121280    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs podName:4cfcfafe-2f29-4161-b095-1f099f33db36 nodeName:}" failed. No retries permitted until 2026-09-16 20:03:00.621266089 +0000 UTC m=+4681.925283033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs") pod "rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 16 20:03:00.625002 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:00.624961    2581 secret.go:189] Couldn't get secret e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 16 20:03:00.625194 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:00.625049    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs podName:4cfcfafe-2f29-4161-b095-1f099f33db36 nodeName:}" failed. No retries permitted until 2026-09-16 20:03:01.625034174 +0000 UTC m=+4682.929051118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs") pod "rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Sep 16 20:03:00.740428 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:00.740382    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:03:01.044948 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:01.044904    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 20:03:01.044948 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:03:01.044948 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 20:03:01.044948 ip-10-0-133-194 kubenswrapper[2581]:  > containerID="c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62" cmd=["/bin/sleep","15"]
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.044970    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" message="" err=<
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]:  >
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:01.044988    2581 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" podUID="4cfcfafe-2f29-4161-b095-1f099f33db36" containerName="main" containerID="cri-o://c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"
Sep 16 20:03:01.045282 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.045043    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" podUID="4cfcfafe-2f29-4161-b095-1f099f33db36" containerName="main" containerID="cri-o://c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62" gracePeriod=60
Sep 16 20:03:01.392521 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.392489    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:03:01.432625 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.432586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-dshm\") pod \"4cfcfafe-2f29-4161-b095-1f099f33db36\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") "
Sep 16 20:03:01.432625 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.432632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-home\") pod \"4cfcfafe-2f29-4161-b095-1f099f33db36\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") "
Sep 16 20:03:01.432891 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.432658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs\") pod \"4cfcfafe-2f29-4161-b095-1f099f33db36\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") "
Sep 16 20:03:01.432891 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.432693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-model-cache\") pod \"4cfcfafe-2f29-4161-b095-1f099f33db36\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") "
Sep 16 20:03:01.432891 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.432732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-tmp-dir\") pod \"4cfcfafe-2f29-4161-b095-1f099f33db36\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") "
Sep 16 20:03:01.433073 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.432977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-home" (OuterVolumeSpecName: "home") pod "4cfcfafe-2f29-4161-b095-1f099f33db36" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:01.433073 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.433008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cfcfafe-2f29-4161-b095-1f099f33db36" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:01.433073 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.433043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cfcfafe-2f29-4161-b095-1f099f33db36" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:01.433237 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.433120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chp4z\" (UniqueName: \"kubernetes.io/projected/4cfcfafe-2f29-4161-b095-1f099f33db36-kube-api-access-chp4z\") pod \"4cfcfafe-2f29-4161-b095-1f099f33db36\" (UID: \"4cfcfafe-2f29-4161-b095-1f099f33db36\") "
Sep 16 20:03:01.433404 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.433376    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:01.433517 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.433406    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:01.433517 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.433421    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:01.435554 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.435532    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-dshm" (OuterVolumeSpecName: "dshm") pod "4cfcfafe-2f29-4161-b095-1f099f33db36" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:01.435660 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.435556    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfcfafe-2f29-4161-b095-1f099f33db36-kube-api-access-chp4z" (OuterVolumeSpecName: "kube-api-access-chp4z") pod "4cfcfafe-2f29-4161-b095-1f099f33db36" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36"). InnerVolumeSpecName "kube-api-access-chp4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:01.435660 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.435649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cfcfafe-2f29-4161-b095-1f099f33db36" (UID: "4cfcfafe-2f29-4161-b095-1f099f33db36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:01.534180 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.534143    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chp4z\" (UniqueName: \"kubernetes.io/projected/4cfcfafe-2f29-4161-b095-1f099f33db36-kube-api-access-chp4z\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:01.534180 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.534174    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cfcfafe-2f29-4161-b095-1f099f33db36-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:01.534180 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:01.534184    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cfcfafe-2f29-4161-b095-1f099f33db36-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:02.044164 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.044127    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cfcfafe-2f29-4161-b095-1f099f33db36" containerID="c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62" exitCode=0
Sep 16 20:03:02.044376 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.044190    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"
Sep 16 20:03:02.044376 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.044221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" event={"ID":"4cfcfafe-2f29-4161-b095-1f099f33db36","Type":"ContainerDied","Data":"c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"}
Sep 16 20:03:02.044376 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.044288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg" event={"ID":"4cfcfafe-2f29-4161-b095-1f099f33db36","Type":"ContainerDied","Data":"122c67273b25c2281320a7f68665161096697b44d2ea4211b0884e143c47d973"}
Sep 16 20:03:02.044376 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.044319    2581 scope.go:117] "RemoveContainer" containerID="c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"
Sep 16 20:03:02.053581 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.053388    2581 scope.go:117] "RemoveContainer" containerID="c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"
Sep 16 20:03:02.053672 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:02.053654    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62\": container with ID starting with c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62 not found: ID does not exist" containerID="c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"
Sep 16 20:03:02.053715 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.053680    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62"} err="failed to get container status \"c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62\": rpc error: code = NotFound desc = could not find container \"c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62\": container with ID starting with c31a833c0c277e498abfee5325bf7e6ac55d8622d23f83aa79ab39959fa40f62 not found: ID does not exist"
Sep 16 20:03:02.070086 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.070060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"]
Sep 16 20:03:02.073718 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:02.073693    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-x7ffg"]
Sep 16 20:03:03.223934 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:03.223901    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cfcfafe-2f29-4161-b095-1f099f33db36" path="/var/lib/kubelet/pods/4cfcfafe-2f29-4161-b095-1f099f33db36/volumes"
Sep 16 20:03:10.739811 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:10.739758    2581 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.97:8000/health\": dial tcp 10.132.0.97:8000: connect: connection refused"
Sep 16 20:03:20.750065 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:20.750032    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:03:20.757683 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:20.757647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:03:25.001427 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:25.001393    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"]
Sep 16 20:03:40.024573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.024514    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main" containerID="cri-o://5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae" gracePeriod=45
Sep 16 20:03:40.669749 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.669722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:03:40.764163 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764069    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tmp-dir\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764163 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-model-cache\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764163 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9c7849d-ccfb-4be8-990e-ad015ff6554b-cabundle-cert\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764440 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764173    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-dshm\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764440 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764197    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-home\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764440 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tls-certs\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764440 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kube-api-access-b6tbf\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764440 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764322    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kserve-provision-location\") pod \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\" (UID: \"a9c7849d-ccfb-4be8-990e-ad015ff6554b\") "
Sep 16 20:03:40.764440 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764384    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:40.764758 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764536    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c7849d-ccfb-4be8-990e-ad015ff6554b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 20:03:40.764758 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764584    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-model-cache\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.764899 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.764851    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-home" (OuterVolumeSpecName: "home") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:40.766568 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.766539    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-dshm" (OuterVolumeSpecName: "dshm") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:40.766820 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.766791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kube-api-access-b6tbf" (OuterVolumeSpecName: "kube-api-access-b6tbf") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "kube-api-access-b6tbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:40.766953 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.766846    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:40.778278 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.778235    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:40.789839 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.789811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9c7849d-ccfb-4be8-990e-ad015ff6554b" (UID: "a9c7849d-ccfb-4be8-990e-ad015ff6554b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:03:40.865790 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865754    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tmp-dir\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.865790 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865785    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a9c7849d-ccfb-4be8-990e-ad015ff6554b-cabundle-cert\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.865790 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865797    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-dshm\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.866037 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865805    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-home\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.866037 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865812    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9c7849d-ccfb-4be8-990e-ad015ff6554b-tls-certs\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.866037 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865820    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6tbf\" (UniqueName: \"kubernetes.io/projected/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kube-api-access-b6tbf\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:40.866037 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:40.865832    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9c7849d-ccfb-4be8-990e-ad015ff6554b-kserve-provision-location\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:03:41.245469 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.245440    2581 generic.go:356] "Generic (PLEG): container finished" podID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerID="5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae" exitCode=0
Sep 16 20:03:41.245799 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.245480    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"
Sep 16 20:03:41.245799 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.245522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" event={"ID":"a9c7849d-ccfb-4be8-990e-ad015ff6554b","Type":"ContainerDied","Data":"5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae"}
Sep 16 20:03:41.245799 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.245564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf" event={"ID":"a9c7849d-ccfb-4be8-990e-ad015ff6554b","Type":"ContainerDied","Data":"5c639c603e9bed44ac558f42811e1118fe262fb69169fc9129da1586103add2d"}
Sep 16 20:03:41.245799 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.245586    2581 scope.go:117] "RemoveContainer" containerID="5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae"
Sep 16 20:03:41.254192 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.254168    2581 scope.go:117] "RemoveContainer" containerID="3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015"
Sep 16 20:03:41.261811 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.261792    2581 scope.go:117] "RemoveContainer" containerID="5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae"
Sep 16 20:03:41.262107 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:41.262089    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae\": container with ID starting with 5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae not found: ID does not exist" containerID="5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae"
Sep 16 20:03:41.262173 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.262114    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae"} err="failed to get container status \"5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae\": rpc error: code = NotFound desc = could not find container \"5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae\": container with ID starting with 5b406c3ae52bca868a198ce79f50b81985b720e7b231d001dba2c61f4d240cae not found: ID does not exist"
Sep 16 20:03:41.262173 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.262130    2581 scope.go:117] "RemoveContainer" containerID="3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015"
Sep 16 20:03:41.262399 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:03:41.262381    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015\": container with ID starting with 3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015 not found: ID does not exist" containerID="3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015"
Sep 16 20:03:41.262477 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.262403    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015"} err="failed to get container status \"3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015\": rpc error: code = NotFound desc = could not find container \"3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015\": container with ID starting with 3656e31990b82d2688c5025c5a0c1fe93e0158145c3631248158966fb5c53015 not found: ID does not exist"
Sep 16 20:03:41.265555 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.265535    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"]
Sep 16 20:03:41.275444 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:41.273686    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-78b56984c6x7fsf"]
Sep 16 20:03:43.221649 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:43.221611    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" path="/var/lib/kubelet/pods/a9c7849d-ccfb-4be8-990e-ad015ff6554b/volumes"
Sep 16 20:03:47.965350 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965315    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26ntz/must-gather-5vj74"]
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965618    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cfcfafe-2f29-4161-b095-1f099f33db36" containerName="main"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965628    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cfcfafe-2f29-4161-b095-1f099f33db36" containerName="main"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="storage-initializer"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965644    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="storage-initializer"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965660    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965666    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965715    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9c7849d-ccfb-4be8-990e-ad015ff6554b" containerName="main"
Sep 16 20:03:47.965723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.965722    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cfcfafe-2f29-4161-b095-1f099f33db36" containerName="main"
Sep 16 20:03:47.969183 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.969165    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:47.971939 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.971912    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26ntz\"/\"default-dockercfg-b86pq\""
Sep 16 20:03:47.972055 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.971916    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26ntz\"/\"kube-root-ca.crt\""
Sep 16 20:03:47.972055 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.971952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26ntz\"/\"openshift-service-ca.crt\""
Sep 16 20:03:47.979076 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:47.979053    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26ntz/must-gather-5vj74"]
Sep 16 20:03:48.118180 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.118144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fszkn\" (UniqueName: \"kubernetes.io/projected/db954119-d09d-4f88-ab1a-3232c33de645-kube-api-access-fszkn\") pod \"must-gather-5vj74\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") " pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.118352 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.118213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db954119-d09d-4f88-ab1a-3232c33de645-must-gather-output\") pod \"must-gather-5vj74\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") " pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.218587 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.218508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fszkn\" (UniqueName: \"kubernetes.io/projected/db954119-d09d-4f88-ab1a-3232c33de645-kube-api-access-fszkn\") pod \"must-gather-5vj74\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") " pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.218587 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.218582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db954119-d09d-4f88-ab1a-3232c33de645-must-gather-output\") pod \"must-gather-5vj74\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") " pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.218960 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.218940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db954119-d09d-4f88-ab1a-3232c33de645-must-gather-output\") pod \"must-gather-5vj74\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") " pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.227401 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.227374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fszkn\" (UniqueName: \"kubernetes.io/projected/db954119-d09d-4f88-ab1a-3232c33de645-kube-api-access-fszkn\") pod \"must-gather-5vj74\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") " pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.279060 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.279031    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:03:48.549443 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.549389    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26ntz/must-gather-5vj74"]
Sep 16 20:03:48.551849 ip-10-0-133-194 kubenswrapper[2581]: W0916 20:03:48.551825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb954119_d09d_4f88_ab1a_3232c33de645.slice/crio-31ac2f99bf3f4282b2898fec936f607f4f91f9be6fdca4781763d58eaac0f4dd WatchSource:0}: Error finding container 31ac2f99bf3f4282b2898fec936f607f4f91f9be6fdca4781763d58eaac0f4dd: Status 404 returned error can't find the container with id 31ac2f99bf3f4282b2898fec936f607f4f91f9be6fdca4781763d58eaac0f4dd
Sep 16 20:03:48.553743 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:48.553722    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:03:49.292532 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:49.292458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26ntz/must-gather-5vj74" event={"ID":"db954119-d09d-4f88-ab1a-3232c33de645","Type":"ContainerStarted","Data":"31ac2f99bf3f4282b2898fec936f607f4f91f9be6fdca4781763d58eaac0f4dd"}
Sep 16 20:03:55.330830 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:55.330727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26ntz/must-gather-5vj74" event={"ID":"db954119-d09d-4f88-ab1a-3232c33de645","Type":"ContainerStarted","Data":"9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d"}
Sep 16 20:03:55.330830 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:55.330784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26ntz/must-gather-5vj74" event={"ID":"db954119-d09d-4f88-ab1a-3232c33de645","Type":"ContainerStarted","Data":"5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"}
Sep 16 20:03:55.351956 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:03:55.351898    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26ntz/must-gather-5vj74" podStartSLOduration=2.215435765 podStartE2EDuration="8.351881068s" totalImagesPullingTime="6.136445303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:03:55.349292059 +0000 UTC m=+4736.653309027" watchObservedRunningTime="2026-09-16 20:03:55.351881068 +0000 UTC m=+4736.655898035"
Sep 16 20:04:11.090012 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:11.089979    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/0.log"
Sep 16 20:04:11.122044 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:11.122003    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/1.log"
Sep 16 20:04:11.136851 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:11.136822    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85465d975b-5frng_7c6728a2-43fd-4f3e-a9e2-acb7789db2a9/router/0.log"
Sep 16 20:04:12.038702 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:12.038649    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/0.log"
Sep 16 20:04:12.072322 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:12.072274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/1.log"
Sep 16 20:04:12.090708 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:12.090679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85465d975b-5frng_7c6728a2-43fd-4f3e-a9e2-acb7789db2a9/router/0.log"
Sep 16 20:04:13.343235 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:13.343198    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-xn9hl_9efff3a4-13c3-4807-8237-fa554431d62e/manager/0.log"
Sep 16 20:04:13.593501 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:13.593430    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8565v_2261aad0-c08c-4c48-8ceb-d452d5b21daf/limitador/0.log"
Sep 16 20:04:13.609758 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:13.609727    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-5xvkz_116b141f-3753-488e-a002-211f731295c8/manager/0.log"
Sep 16 20:04:16.446520 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:16.446490    2581 generic.go:356] "Generic (PLEG): container finished" podID="db954119-d09d-4f88-ab1a-3232c33de645" containerID="5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833" exitCode=0
Sep 16 20:04:16.447128 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:16.446558    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26ntz/must-gather-5vj74" event={"ID":"db954119-d09d-4f88-ab1a-3232c33de645","Type":"ContainerDied","Data":"5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"}
Sep 16 20:04:16.447128 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:16.446844    2581 scope.go:117] "RemoveContainer" containerID="5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"
Sep 16 20:04:16.482132 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:16.482108    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-26ntz_must-gather-5vj74_db954119-d09d-4f88-ab1a-3232c33de645/gather/0.log"
Sep 16 20:04:20.295236 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:20.295201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hxpgk_6f52e1e8-f991-4bb7-8e84-1043fb9d5ab9/global-pull-secret-syncer/0.log"
Sep 16 20:04:20.438306 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:20.438275    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gxggm_f52644df-0002-449f-8b36-bc5a51714e1a/konnectivity-agent/0.log"
Sep 16 20:04:20.520589 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:20.520555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-194.ec2.internal_bee55f8a1340335ec8a8808d97d355d4/haproxy/0.log"
Sep 16 20:04:21.924781 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:21.924699    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-26ntz/must-gather-5vj74"]
Sep 16 20:04:21.925173 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:21.924909    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-26ntz/must-gather-5vj74" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="copy" containerID="cri-o://9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d" gracePeriod=2
Sep 16 20:04:21.927516 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:04:21.927482    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5vj74\" is forbidden: User \"system:node:ip-10-0-133-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-26ntz\": no relationship found between node 'ip-10-0-133-194.ec2.internal' and this object" podUID="db954119-d09d-4f88-ab1a-3232c33de645" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:04:21.929442 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:21.929303    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-26ntz/must-gather-5vj74"]
Sep 16 20:04:22.154450 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.154426    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-26ntz_must-gather-5vj74_db954119-d09d-4f88-ab1a-3232c33de645/copy/0.log"
Sep 16 20:04:22.156137 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.156117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:04:22.158314 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:04:22.158290    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5vj74\" is forbidden: User \"system:node:ip-10-0-133-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-26ntz\": no relationship found between node 'ip-10-0-133-194.ec2.internal' and this object" podUID="db954119-d09d-4f88-ab1a-3232c33de645" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:04:22.200384 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.200318    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fszkn\" (UniqueName: \"kubernetes.io/projected/db954119-d09d-4f88-ab1a-3232c33de645-kube-api-access-fszkn\") pod \"db954119-d09d-4f88-ab1a-3232c33de645\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") "
Sep 16 20:04:22.200384 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.200374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db954119-d09d-4f88-ab1a-3232c33de645-must-gather-output\") pod \"db954119-d09d-4f88-ab1a-3232c33de645\" (UID: \"db954119-d09d-4f88-ab1a-3232c33de645\") "
Sep 16 20:04:22.202494 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.202472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db954119-d09d-4f88-ab1a-3232c33de645-kube-api-access-fszkn" (OuterVolumeSpecName: "kube-api-access-fszkn") pod "db954119-d09d-4f88-ab1a-3232c33de645" (UID: "db954119-d09d-4f88-ab1a-3232c33de645"). InnerVolumeSpecName "kube-api-access-fszkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:04:22.207307 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.207282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db954119-d09d-4f88-ab1a-3232c33de645-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "db954119-d09d-4f88-ab1a-3232c33de645" (UID: "db954119-d09d-4f88-ab1a-3232c33de645"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 20:04:22.301323 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.301288    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fszkn\" (UniqueName: \"kubernetes.io/projected/db954119-d09d-4f88-ab1a-3232c33de645-kube-api-access-fszkn\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:04:22.301323 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.301318    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/db954119-d09d-4f88-ab1a-3232c33de645-must-gather-output\") on node \"ip-10-0-133-194.ec2.internal\" DevicePath \"\""
Sep 16 20:04:22.480592 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.480515    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-26ntz_must-gather-5vj74_db954119-d09d-4f88-ab1a-3232c33de645/copy/0.log"
Sep 16 20:04:22.480953 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.480929    2581 generic.go:356] "Generic (PLEG): container finished" podID="db954119-d09d-4f88-ab1a-3232c33de645" containerID="9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d" exitCode=143
Sep 16 20:04:22.481025 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.480979    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:04:22.481025 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.481018    2581 scope.go:117] "RemoveContainer" containerID="9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d"
Sep 16 20:04:22.483244 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:04:22.483215    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5vj74\" is forbidden: User \"system:node:ip-10-0-133-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-26ntz\": no relationship found between node 'ip-10-0-133-194.ec2.internal' and this object" podUID="db954119-d09d-4f88-ab1a-3232c33de645" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:04:22.489396 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.489292    2581 scope.go:117] "RemoveContainer" containerID="5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"
Sep 16 20:04:22.492037 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:04:22.492011    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5vj74\" is forbidden: User \"system:node:ip-10-0-133-194.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-26ntz\": no relationship found between node 'ip-10-0-133-194.ec2.internal' and this object" podUID="db954119-d09d-4f88-ab1a-3232c33de645" pod="openshift-must-gather-26ntz/must-gather-5vj74"
Sep 16 20:04:22.502811 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.502791    2581 scope.go:117] "RemoveContainer" containerID="9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d"
Sep 16 20:04:22.503095 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:04:22.503075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d\": container with ID starting with 9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d not found: ID does not exist" containerID="9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d"
Sep 16 20:04:22.503152 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.503103    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d"} err="failed to get container status \"9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d\": rpc error: code = NotFound desc = could not find container \"9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d\": container with ID starting with 9af98d7aa3e4596359e6d4911524f748ac063fea57b0e6d1afd2a3a4c313691d not found: ID does not exist"
Sep 16 20:04:22.503152 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.503119    2581 scope.go:117] "RemoveContainer" containerID="5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"
Sep 16 20:04:22.503354 ip-10-0-133-194 kubenswrapper[2581]: E0916 20:04:22.503335    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833\": container with ID starting with 5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833 not found: ID does not exist" containerID="5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"
Sep 16 20:04:22.503395 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:22.503360    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833"} err="failed to get container status \"5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833\": rpc error: code = NotFound desc = could not find container \"5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833\": container with ID starting with 5f93f2a4bb076e4a040f8f1ed518d91977f7975681a2f5e8c2417d7320648833 not found: ID does not exist"
Sep 16 20:04:23.222151 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:23.222115    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db954119-d09d-4f88-ab1a-3232c33de645" path="/var/lib/kubelet/pods/db954119-d09d-4f88-ab1a-3232c33de645/volumes"
Sep 16 20:04:25.991015 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:25.990982    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-xn9hl_9efff3a4-13c3-4807-8237-fa554431d62e/manager/0.log"
Sep 16 20:04:26.299180 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:26.299089    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-8565v_2261aad0-c08c-4c48-8ceb-d452d5b21daf/limitador/0.log"
Sep 16 20:04:26.331592 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:26.331555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-5xvkz_116b141f-3753-488e-a002-211f731295c8/manager/0.log"
Sep 16 20:04:28.141034 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:28.140994    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzvlr_4bf8bb78-8dce-4a5d-84fa-803067c5944c/node-exporter/0.log"
Sep 16 20:04:28.166282 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:28.166244    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzvlr_4bf8bb78-8dce-4a5d-84fa-803067c5944c/kube-rbac-proxy/0.log"
Sep 16 20:04:28.197336 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:28.197315    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mzvlr_4bf8bb78-8dce-4a5d-84fa-803067c5944c/init-textfile/0.log"
Sep 16 20:04:30.825290 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:30.825256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/2.log"
Sep 16 20:04:30.832895 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:30.832868    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-krl58_b1c6d5ee-8126-4260-88a3-7920cf4cb532/console-operator/3.log"
Sep 16 20:04:31.046411 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046380    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"]
Sep 16 20:04:31.046698 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046686    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="copy"
Sep 16 20:04:31.046743 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046700    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="copy"
Sep 16 20:04:31.046743 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046720    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="gather"
Sep 16 20:04:31.046743 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046726    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="gather"
Sep 16 20:04:31.046840 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046779    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="copy"
Sep 16 20:04:31.046840 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.046789    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db954119-d09d-4f88-ab1a-3232c33de645" containerName="gather"
Sep 16 20:04:31.049527 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.049507    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.052758 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.052740    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zj9p\"/\"openshift-service-ca.crt\""
Sep 16 20:04:31.054166 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.054148    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9zj9p\"/\"kube-root-ca.crt\""
Sep 16 20:04:31.054410 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.054178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9zj9p\"/\"default-dockercfg-9qldv\""
Sep 16 20:04:31.059898 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.059874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"]
Sep 16 20:04:31.166842 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.166813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-proc\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.166995 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.166850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-lib-modules\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.166995 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.166928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-sys\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.166995 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.166950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6wj\" (UniqueName: \"kubernetes.io/projected/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-kube-api-access-gb6wj\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.166995 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.166979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-podres\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267419 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-lib-modules\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-sys\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6wj\" (UniqueName: \"kubernetes.io/projected/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-kube-api-access-gb6wj\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-podres\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-proc\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-sys\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267790 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-proc\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267790 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-lib-modules\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.267790 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.267648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-podres\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.276014 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.275996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6wj\" (UniqueName: \"kubernetes.io/projected/e1f3b12a-a6b4-46b5-8429-c350f1d710c6-kube-api-access-gb6wj\") pod \"perf-node-gather-daemonset-88swq\" (UID: \"e1f3b12a-a6b4-46b5-8429-c350f1d710c6\") " pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.359784 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.359761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:31.630718 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:31.630690    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"]
Sep 16 20:04:31.632716 ip-10-0-133-194 kubenswrapper[2581]: W0916 20:04:31.632689    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode1f3b12a_a6b4_46b5_8429_c350f1d710c6.slice/crio-86baf4832d63aae8c63e779be19afc94ff88acd1f936246e381264e8e8bd0204 WatchSource:0}: Error finding container 86baf4832d63aae8c63e779be19afc94ff88acd1f936246e381264e8e8bd0204: Status 404 returned error can't find the container with id 86baf4832d63aae8c63e779be19afc94ff88acd1f936246e381264e8e8bd0204
Sep 16 20:04:32.534335 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:32.534300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq" event={"ID":"e1f3b12a-a6b4-46b5-8429-c350f1d710c6","Type":"ContainerStarted","Data":"363c7880147525db65f6d4c5270427ef600439db50f4a14dec06812125e6b367"}
Sep 16 20:04:32.534723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:32.534360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq" event={"ID":"e1f3b12a-a6b4-46b5-8429-c350f1d710c6","Type":"ContainerStarted","Data":"86baf4832d63aae8c63e779be19afc94ff88acd1f936246e381264e8e8bd0204"}
Sep 16 20:04:32.534723 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:32.534448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:32.551716 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:32.551673    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq" podStartSLOduration=1.5516601859999999 podStartE2EDuration="1.551660186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:04:32.549936128 +0000 UTC m=+4773.853953095" watchObservedRunningTime="2026-09-16 20:04:32.551660186 +0000 UTC m=+4773.855677152"
Sep 16 20:04:33.097191 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:33.097161    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkbvg_5232b259-8d8e-4a9e-bbec-26720ec51c17/dns/0.log"
Sep 16 20:04:33.118975 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:33.118949    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pkbvg_5232b259-8d8e-4a9e-bbec-26720ec51c17/kube-rbac-proxy/0.log"
Sep 16 20:04:33.223468 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:33.223434    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ptxgf_3780632e-fea7-4460-a811-203997052dcb/dns-node-resolver/0.log"
Sep 16 20:04:33.913552 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:33.913515    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-67d58_15e98837-18f9-4305-a42e-695792e504f2/node-ca/0.log"
Sep 16 20:04:35.123364 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:35.123310    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/0.log"
Sep 16 20:04:35.151573 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:35.151529    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/1.log"
Sep 16 20:04:35.177819 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:35.177788    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85465d975b-5frng_7c6728a2-43fd-4f3e-a9e2-acb7789db2a9/router/0.log"
Sep 16 20:04:35.699080 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:35.699047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hwz4m_fab74c7b-6b13-4e5e-a494-93793cda1407/serve-healthcheck-canary/0.log"
Sep 16 20:04:36.259638 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:36.259606    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9sc4w_9c6bdbcf-9c07-47b1-adcd-9a63a682509a/kube-rbac-proxy/0.log"
Sep 16 20:04:36.288604 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:36.288528    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9sc4w_9c6bdbcf-9c07-47b1-adcd-9a63a682509a/exporter/0.log"
Sep 16 20:04:36.311075 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:36.311043    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9sc4w_9c6bdbcf-9c07-47b1-adcd-9a63a682509a/extractor/0.log"
Sep 16 20:04:38.548852 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:38.548820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9zj9p/perf-node-gather-daemonset-88swq"
Sep 16 20:04:40.404550 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:40.404520    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-764fc98f48-d58bz_dd94dacb-852f-4d4b-9c99-4736fa00044e/manager/0.log"
Sep 16 20:04:41.096392 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:41.096302    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-b684f68f9-5p4wb_19a6466e-fe4a-4fd1-9016-d6d4eb73c5ab/manager/0.log"
Sep 16 20:04:41.119049 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:41.119026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-6kmjf_86ef3166-3d20-4dd4-85d4-3b10cbc00bfd/server/0.log"
Sep 16 20:04:41.533512 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:41.533482    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/0.log"
Sep 16 20:04:41.570963 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:41.570928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/1.log"
Sep 16 20:04:41.588117 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:41.588073    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85465d975b-5frng_7c6728a2-43fd-4f3e-a9e2-acb7789db2a9/router/0.log"
Sep 16 20:04:42.660704 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.660632    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/0.log"
Sep 16 20:04:42.701552 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.701485    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-5p4nm_9c6de2c0-b034-4e27-8331-8113e71873a5/istio-proxy/1.log"
Sep 16 20:04:42.719497 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.719461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85465d975b-5frng_7c6728a2-43fd-4f3e-a9e2-acb7789db2a9/router/0.log"
Sep 16 20:04:42.861295 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.861257    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-nv2qk_4ca058e9-7f14-41fb-b0d3-3ba5d19b0855/manager/0.log"
Sep 16 20:04:42.906112 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.906084    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cxp59_58139ed6-7b53-4870-aa73-6bbf9cfb33e1/s3-init/0.log"
Sep 16 20:04:42.925848 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.925825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-cxp59_58139ed6-7b53-4870-aa73-6bbf9cfb33e1/download-hf-model/0.log"
Sep 16 20:04:42.959746 ip-10-0-133-194 kubenswrapper[2581]: I0916 20:04:42.959707    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-b7rzb_ee5a6432-9b17-462e-84a4-48319d5bca5a/seaweedfs/0.log"