Sep 19 08:18:39.248887 ip-10-0-140-191 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 19 08:18:39.248898 ip-10-0-140-191 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 19 08:18:39.248908 ip-10-0-140-191 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 19 08:18:39.249116 ip-10-0-140-191 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 19 08:18:49.336948 ip-10-0-140-191 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 19 08:18:49.336969 ip-10-0-140-191 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7a45af8813ca48a5b9d59c7a55c350c9 --
Sep 19 08:20:46.319525 ip-10-0-140-191 systemd[1]: Starting Kubernetes Kubelet...
Sep 19 08:20:49.888699 ip-10-0-140-191 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:49.888699 ip-10-0-140-191 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 19 08:20:49.888699 ip-10-0-140-191 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:49.888699 ip-10-0-140-191 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 19 08:20:49.889501 ip-10-0-140-191 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 19 08:20:49.889501 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.888817    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 19 08:20:49.893072 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893055    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.893072 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893072    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893078    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893082    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893085    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893088    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893091    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893093    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893096    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893099    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893101    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893104    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893106    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893109    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893116    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893119    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893122    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893124    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893127    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893129    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893132    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.893138 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893134    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893137    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893140    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893143    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893146    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893149    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893152    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893155    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893157    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893160    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893162    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893165    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893168    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893170    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893174    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893178    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893181    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893184    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893186    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893189    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.893659 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893191    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893194    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893196    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893199    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893203    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893205    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893208    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893210    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893212    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893215    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893232    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893236    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893238    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893240    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893244    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893247    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893249    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893252    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893255    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.894143 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893257    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893261    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893265    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893268    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893270    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893273    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893275    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893278    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893281    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893283    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893286    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893289    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893292    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893295    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893298    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893301    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893304    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893306    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893309    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893312    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.894617 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893314    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893317    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893319    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893322    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893324    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893327    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893329    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893332    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893335    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893338    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893341    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893344    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893346    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893349    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893352    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893354    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893357    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893360    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.893362    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894033    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.895103 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894040    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894043    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894046    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894049    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894052    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894055    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894057    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894060    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894063    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894066    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894069    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894072    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894074    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894077    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894079    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894082    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894084    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894087    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894090    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.895597 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894094    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894096    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894099    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894102    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894105    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894107    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894110    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894113    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894115    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894118    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894121    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894123    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894126    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894128    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894131    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894134    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894136    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894139    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894142    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894145    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.896109 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894148    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894151    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894153    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894156    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894159    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894162    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894165    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894167    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894170    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894173    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894175    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894179    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894183    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894187    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894190    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894193    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894196    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894199    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894202    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.896606 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894205    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894207    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894211    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894215    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894232    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894235    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894238    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894240    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894242    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894245    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894248    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894250    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894253    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894256    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894259    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894261    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894264    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894267    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894270    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894272    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.897080 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894275    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894278    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894280    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894283    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894286    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894289    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894291    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894294    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894296    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894299    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894301    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894304    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894307    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894310    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894312    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894316    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894319    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894322    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894324    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894327    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.897577 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894683    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894687    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894690    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894693    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894697    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894699    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894703    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894706    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894709    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894712    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894715    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894718    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894721    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894724    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894727    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894729    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894732    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894734    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894737    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.898050 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894740    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894742    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894744    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894747    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894749    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894752    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894754    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894757    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894759    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894762    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894764    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894767    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894770    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894772    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894774    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894777    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894779    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894782    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894785    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894788    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.898517 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894791    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894793    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894796    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894798    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894801    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894804    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894807    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894810    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894812    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894815    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894817    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894820    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894822    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894825    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894829    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894832    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894836    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894839    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894841    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.899043 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894844    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894847    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894850    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894852    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894856    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894858    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894861    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894863    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894866    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894868    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894871    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894874    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894877    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894880    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894883    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894886    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894889    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894891    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894894    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894896    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.899508 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894899    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894903    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894906    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894908    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894911    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894914    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894917    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894920    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894922    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894925    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894927    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894930    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894932    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894935    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894937    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894940    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894943    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894945    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894948    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.899992 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894950    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.894952    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895028    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895071    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895080    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895086    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895091    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895095    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895099    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895103    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895107    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895110    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895114    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895117    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895121    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895124    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895127    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895130    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895133    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895136    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895140    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895143    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895146    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895149    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 19 08:20:49.900460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895152    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895156    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895159    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895162    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895166    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895169    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895172    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895175    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895179    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895181    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895185    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895188    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895194    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895198    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895243    2580 flags.go:64] FLAG: --enable-server="true"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895249    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895254    2580 flags.go:64] FLAG: --event-burst="100"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895258    2580 flags.go:64] FLAG: --event-qps="50"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895261    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895264    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895267    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895271    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895273    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895277    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895280    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 19 08:20:49.901079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895283    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895286    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895289    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895292    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895295    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895298    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895301    2580 flags.go:64] FLAG: --feature-gates=""
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895304    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895307    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895310    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895314    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895317    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895320    2580 flags.go:64] FLAG: --help="false"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895323    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-191.ec2.internal"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895326    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895329    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895332    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895335    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895339    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895343    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895347    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895350    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895353    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 19 08:20:49.901779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895356    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895360    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895363    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895366    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895369    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895372    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895374    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895377    2580 flags.go:64] FLAG: --lock-file=""
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895380    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895383    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895386    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895391    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895394    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895396    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895399    2580 flags.go:64] FLAG: --logging-format="text"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895402    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895405    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895408    2580 flags.go:64] FLAG: --manifest-url=""
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895411    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895415    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895419    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895422    2580 flags.go:64] FLAG: --max-pods="110"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895425    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895428    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895431    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 19 08:20:49.902803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895434    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895437    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895440    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895444    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895454    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895458    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895461    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895464    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895469    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895475    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895478    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895481    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895485    2580 flags.go:64] FLAG: --port="10250"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895488    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895490    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-034e70e5fe901b056"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895494    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895496    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895499    2580 flags.go:64] FLAG: --register-node="true"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895502    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895506    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895509    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895512    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895514    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895518    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895521    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 19 08:20:49.903868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895524    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895527    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895530    2580 flags.go:64] FLAG: --runonce="false"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895532    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895535    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895538    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895541    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895544    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895547    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895550    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895554    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895557    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895561    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895563    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895566    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895569    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895573    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895575    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895581    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895583    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895586    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895590    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895593    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895596    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895599    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 19 08:20:49.904968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895602    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895605    2580 flags.go:64] FLAG: --v="2"
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895609    2580 flags.go:64] FLAG: --version="false"
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895613    2580 flags.go:64] FLAG: --vmodule=""
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895617    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.895621    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895732    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895736    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895739    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895743    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895746    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895750    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895752    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895755    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895759    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895762    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895766    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895770    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895773    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895778    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895781    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.905929 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895784    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895787    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895789    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895798    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895801    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895804    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895807    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895809    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895812    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895815    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895817    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895820    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895823    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895826    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895828    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895831    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895833    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895836    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895838    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895841    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.906742 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895843    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895846    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895849    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895851    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895854    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895856    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895859    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895861    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895867    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895870    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895874    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895877    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895880    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895883    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895886    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895889    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895897    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895900    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895902    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.907586 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895905    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895907    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895910    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895913    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895915    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895918    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895920    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895923    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895926    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895929    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895932    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895934    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895937    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895939    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895943    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895947    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895949    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895952    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895955    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.908104 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895957    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895960    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895964    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895966    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895971    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895973    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895976    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895979    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895981    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895984    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895986    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895989    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895992    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895994    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895997    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.895999    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896002    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896004    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896007    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.908808 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896010    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896012    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896015    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896017    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896020    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896022    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.896025    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.896468    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.904166    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.904184    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.904208    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.904240    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904334    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904342    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904347    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.909329 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904352    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904357    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904362    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904367    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904372    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904376    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904381    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904386    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904390    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904394    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904399    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904403    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904408    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904413    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904417    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904421    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904425    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904430    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904435    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904439    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.909709 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904443    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904447    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904452    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904456    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904460    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904464    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904471    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904475    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904480    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904484    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904488    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904495    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904501    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904505    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904513    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904519    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904524    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904529    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904533    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.910528 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904538    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904542    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904546    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904551    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904555    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904559    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904563    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904568    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904572    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904576    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904580    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904584    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904588    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904593    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904597    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904601    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904605    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904609    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904614    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.911212 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904621    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904628    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904632    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904638    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904643    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904647    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904652    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904657    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904662    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904666    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904670    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904674    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904679    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904683    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904687    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904691    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904696    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904700    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904704    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904708    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.911734 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904712    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904716    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904721    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904725    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904729    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904733    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904737    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904742    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904746    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904750    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904754    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904758    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904762    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904766    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904771    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904776    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904780    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.912248 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904784    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.904792    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.904990    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905000    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905005    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905010    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905014    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905018    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905023    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905027    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905031    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905035    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905039    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905043    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905048    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 19 08:20:49.912667 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905052    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905057    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905061    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905065    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905069    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905074    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905078    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905082    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905087    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905091    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905096    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905100    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905104    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905108    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905112    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905117    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905122    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905126    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905130    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 19 08:20:49.913039 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905135    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905140    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905147    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905151    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905156    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905160    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905164    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905169    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905173    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905178    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905182    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905186    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905190    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905194    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905198    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905202    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905206    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905210    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905215    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 19 08:20:49.913539 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905237    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905242    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905246    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905250    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905254    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905259    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905263    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905267    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905273    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905279    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905285    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905290    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905295    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905299    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905303    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905307    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905311    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905315    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905319    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 19 08:20:49.914010 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905323    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905327    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905331    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905336    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905340    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905345    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905349    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905355    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905359    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905366    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905371    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905376    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905381    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905385    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905389    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905393    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905397    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905401    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905406    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 19 08:20:49.925619 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905410    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905414    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905419    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905423    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905427    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905432    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905436    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905441    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905445    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:49.905449    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.905457    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.905604    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.908134    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.909019    2580 server.go:1019] "Starting client certificate rotation"
Sep 19 08:20:49.926145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.909124    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:49.926559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.909160    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 19 08:20:49.926559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.912407    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:49.926559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.913532    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 19 08:20:49.928679 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.928664    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 19 08:20:49.930293 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.930212    2580 log.go:25] "Validated CRI v1 image API"
Sep 19 08:20:49.931090 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.931075    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 19 08:20:49.933752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.933735    2580 fs.go:135] Filesystem UUIDs: map[5cb88782-74d6-4c50-9d97-6dab4931b9e2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e9763880-df95-4935-b4fb-40c27bd8d019:/dev/nvme0n1p4]
Sep 19 08:20:49.933801 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.933752    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 19 08:20:49.937525 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.937400    2580 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:49.936904951 +0000 UTC m=+2.123096901 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102788 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2995651862bfd67dfe176960d4d6b9 SystemUUID:ec299565-1862-bfd6-7dfe-176960d4d6b9 BootID:7a45af88-13ca-48a5-b9d5-9c7a55c350c9 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:c7:ea:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:c7:ea:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:d0:ab:1f:08:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 19 08:20:49.937525 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.937518    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 19 08:20:49.937645 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.937620    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 19 08:20:49.938927 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.938904    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 19 08:20:49.939062 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.938930    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 19 08:20:49.940038 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.940028    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 19 08:20:49.940076 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.940041    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 19 08:20:49.940076 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.940056    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:49.940076 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.940067    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 19 08:20:49.940155 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.940139    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 19 08:20:49.943603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943589    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:49.943710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943701    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 19 08:20:49.943752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943733    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 19 08:20:49.943752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943742    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 19 08:20:49.943817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943756    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 19 08:20:49.943817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943764    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 19 08:20:49.943817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.943773    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 19 08:20:49.944576 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.944561    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:49.944668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.944584    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 19 08:20:49.955142 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.955120    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 19 08:20:49.957189 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.957174    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 19 08:20:49.957268 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.957206    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 19 08:20:49.957685 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.957671    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:49.958168 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958156    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 19 08:20:49.958209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958173    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 19 08:20:49.958209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958180    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 19 08:20:49.958209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958189    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 19 08:20:49.958209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958200    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 19 08:20:49.958209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958207    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958240    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958247    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958255    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958260    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958268    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958275    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958298    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 19 08:20:49.958353 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.958304    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 19 08:20:49.960066 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.960046    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 19 08:20:49.960127 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.960048    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 19 08:20:49.961998 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.961987    2580 server.go:1268] "Started kubelet"
Sep 19 08:20:49.962109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.962085    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 19 08:20:49.962183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.962074    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 19 08:20:49.962259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.962212    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 19 08:20:49.962857 ip-10-0-140-191 systemd[1]: Started Kubernetes Kubelet.
Sep 19 08:20:49.963274 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.963259    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 19 08:20:49.963686 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.963496    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 19 08:20:49.964236 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.964204    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 19 08:20:49.966543 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.966526    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 19 08:20:49.968476 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.968459    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 19 08:20:49.968572 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.968479    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:49.968572 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.968540    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:49.969002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.968982    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 19 08:20:49.969002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969003    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 19 08:20:49.969211 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969194    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 19 08:20:49.969211 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969203    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 19 08:20:49.969342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969238    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 19 08:20:49.969704 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969690    2580 factory.go:153] Registering CRI-O factory
Sep 19 08:20:49.969777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969706    2580 factory.go:223] Registration of the crio container factory successfully
Sep 19 08:20:49.969821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969793    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 19 08:20:49.969821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969802    2580 factory.go:55] Registering systemd factory
Sep 19 08:20:49.969821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969807    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 19 08:20:49.969821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969821    2580 factory.go:103] Registering Raw factory
Sep 19 08:20:49.969973 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.969830    2580 manager.go:1196] Started watching for new ooms in manager
Sep 19 08:20:49.970206 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.970196    2580 manager.go:349] Starting recovery of all containers
Sep 19 08:20:49.971569 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.971546    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 19 08:20:49.971652 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.971610    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 19 08:20:49.977799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.977774    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbq4j"
Sep 19 08:20:49.991799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.991773    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbq4j"
Sep 19 08:20:49.992678 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.992658    2580 manager.go:354] Recovery completed
Sep 19 08:20:49.995915 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:49.995047    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-191.ec2.internal.18d6ab1ea0d3edab  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-191.ec2.internal,UID:ip-10-0-140-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-191.ec2.internal,},FirstTimestamp:2026-09-19 08:20:49.961962923 +0000 UTC m=+2.148154870,LastTimestamp:2026-09-19 08:20:49.961962923 +0000 UTC m=+2.148154870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-191.ec2.internal,}"
Sep 19 08:20:49.997603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.997590    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.000065 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.999442    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.000065 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.999469    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.000065 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:49.999483    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.000394 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.000383    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.000477 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.000460    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 19 08:20:50.000537 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.000477    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 19 08:20:50.000537 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.000495    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 19 08:20:50.001141 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.001126    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.001201 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.001153    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.001201 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.001167    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.002351 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.002335    2580 policy_none.go:49] "None policy: Start"
Sep 19 08:20:50.002351 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.002351    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 19 08:20:50.002476 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.002361    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 19 08:20:50.004257 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.004244    2580 policy_none.go:47] "Start"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.048737    2580 manager.go:337] "Starting Device Plugin manager"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.048776    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.048790    2580 server.go:87] "Starting device plugin registration server"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.049049    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.049061    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.049129    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.049204    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.049213    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.049766    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 19 08:20:50.054763 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.049813    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.072021 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.071990    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 19 08:20:50.073252 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.073234    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 19 08:20:50.073329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.073283    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 19 08:20:50.073329 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.073307    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 19 08:20:50.073391 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.073354    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 19 08:20:50.075792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.075776    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:50.149244 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.149153    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.150077 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.150062    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.150145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.150092    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.150145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.150107    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.150145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.150132    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.158514 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.158490    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.158514 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.158512    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-191.ec2.internal\": node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.173431 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.173396    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"]
Sep 19 08:20:50.173538 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.173456    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.173538 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.173476    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.174248 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.174211    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.174333 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.174254    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.174333 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.174265    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.176359 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.176346    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.176470 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.176456    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.176504 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.176491    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.177039 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.177022    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.177113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.177040    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.177113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.177052    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.177113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.177062    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.177113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.177072    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.177113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.177075    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.179241 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.179214    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.179314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.179251    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 19 08:20:50.181545 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.181531    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientMemory"
Sep 19 08:20:50.181623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.181554    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasNoDiskPressure"
Sep 19 08:20:50.181623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.181564    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeHasSufficientPID"
Sep 19 08:20:50.192731 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.192715    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-191.ec2.internal\" not found" node="ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.197928 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.197914    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-191.ec2.internal\" not found" node="ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.271341 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.271320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78b47eda2825784ab15cc01c09e532f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal\" (UID: \"78b47eda2825784ab15cc01c09e532f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.271441 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.271344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b47eda2825784ab15cc01c09e532f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal\" (UID: \"78b47eda2825784ab15cc01c09e532f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.271441 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.271364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b28b83e27db47df5e96b9958be391357-config\") pod \"kube-apiserver-proxy-ip-10-0-140-191.ec2.internal\" (UID: \"b28b83e27db47df5e96b9958be391357\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.274403 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.274387    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.372161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.372134    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78b47eda2825784ab15cc01c09e532f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal\" (UID: \"78b47eda2825784ab15cc01c09e532f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.372289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.372163    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b47eda2825784ab15cc01c09e532f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal\" (UID: \"78b47eda2825784ab15cc01c09e532f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.372289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.372185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b28b83e27db47df5e96b9958be391357-config\") pod \"kube-apiserver-proxy-ip-10-0-140-191.ec2.internal\" (UID: \"b28b83e27db47df5e96b9958be391357\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.372289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.372241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78b47eda2825784ab15cc01c09e532f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal\" (UID: \"78b47eda2825784ab15cc01c09e532f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.372289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.372251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78b47eda2825784ab15cc01c09e532f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal\" (UID: \"78b47eda2825784ab15cc01c09e532f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.372424 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.372302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b28b83e27db47df5e96b9958be391357-config\") pod \"kube-apiserver-proxy-ip-10-0-140-191.ec2.internal\" (UID: \"b28b83e27db47df5e96b9958be391357\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.375261 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.375245    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.475570 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.475544    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.495712 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.495682    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.501918 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.501898    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:50.576275 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.576245    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.676744 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.676715    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.777367 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.777290    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.877791 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.877765    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.908962 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.908938    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 19 08:20:50.909384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.909072    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 19 08:20:50.978434 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:50.978408    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:50.996589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.996535    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:49 +0000 UTC" deadline="2028-05-16 19:55:08.685034131 +0000 UTC"
Sep 19 08:20:50.996589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:50.996588    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14531h34m17.688454616s"
Sep 19 08:20:51.079445 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.079256    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:51.179772 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.179745    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:51.181645 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:51.181614    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b47eda2825784ab15cc01c09e532f6.slice/crio-9168cc5ccf49e7178a46c1c4fd5eadaf829f31dc8be353f4d5c7d26d8a609c4a WatchSource:0}: Error finding container 9168cc5ccf49e7178a46c1c4fd5eadaf829f31dc8be353f4d5c7d26d8a609c4a: Status 404 returned error can't find the container with id 9168cc5ccf49e7178a46c1c4fd5eadaf829f31dc8be353f4d5c7d26d8a609c4a
Sep 19 08:20:51.183396 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:51.183365    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28b83e27db47df5e96b9958be391357.slice/crio-60fa7622b9ca2dc57b4ea30d0f5046ed4e2277f828c631af0f99bdd06c87ff32 WatchSource:0}: Error finding container 60fa7622b9ca2dc57b4ea30d0f5046ed4e2277f828c631af0f99bdd06c87ff32: Status 404 returned error can't find the container with id 60fa7622b9ca2dc57b4ea30d0f5046ed4e2277f828c631af0f99bdd06c87ff32
Sep 19 08:20:51.188303 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.188288    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:20:51.251980 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.251951    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.280246 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.280185    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:51.380710 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.380646    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-191.ec2.internal\" not found"
Sep 19 08:20:51.427951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.427928    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.433257 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.433234    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.469316 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.469294    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:51.480770 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.480752    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:51.481720 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.481704    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal"
Sep 19 08:20:51.489699 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.489685    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 19 08:20:51.944320 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.944292    2580 apiserver.go:52] "Watching apiserver"
Sep 19 08:20:51.951895 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.951868    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 19 08:20:51.952263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.952236    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-dkxrb","openshift-network-diagnostics/network-check-target-jnc2h","openshift-ovn-kubernetes/ovnkube-node-78n4r","kube-system/global-pull-secret-syncer-ttzbh","kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4","openshift-cluster-node-tuning-operator/tuned-j8xz4","openshift-dns/node-resolver-q55gs","openshift-multus/multus-additional-cni-plugins-xtf4w","openshift-network-operator/iptables-alerter-xlzz6","kube-system/konnectivity-agent-m87q2","openshift-image-registry/node-ca-mpps7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal","openshift-multus/multus-whtmv"]
Sep 19 08:20:51.955015 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.954823    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.956975 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.956952    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:51.957848 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.957622    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.957954 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.957861    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qnks7\""
Sep 19 08:20:51.958330 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.958106    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:51.959300 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.959277    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.959404 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.959334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nj9vm\""
Sep 19 08:20:51.959463 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.959401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.960357 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.960152    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 19 08:20:51.962016 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.961995    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:51.962130 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.962068    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:20:51.963048 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.962873    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 19 08:20:51.963574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.963555    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.963667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.963608    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 19 08:20:51.963852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.963834    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 19 08:20:51.963900 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.963890    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 19 08:20:51.964275 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.964100    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 19 08:20:51.965270 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.965173    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:51.965270 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.965257    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:20:51.965430 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.965375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95pdx\""
Sep 19 08:20:51.968279 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.968259    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.968944 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.968923    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 19 08:20:51.969018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.968927    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 19 08:20:51.970526 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.970343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.970526 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.970439    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2ljvz\""
Sep 19 08:20:51.970526 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.970468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 19 08:20:51.970889 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.970869    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.971429 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.971409    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 19 08:20:51.973094 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.973078    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 19 08:20:51.973291 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.973270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-whtmv"
Sep 19 08:20:51.973489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.973465    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.974068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.973704    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 19 08:20:51.974068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.973897    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ztssn\""
Sep 19 08:20:51.974068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.973920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 19 08:20:51.974068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.974064    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 19 08:20:51.975442 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.975417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jstl7\""
Sep 19 08:20:51.975542 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.975434    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 19 08:20:51.975913 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.975894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:51.976000 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:51.975951    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:20:51.978986 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.978968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:51.981060 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-etc-selinux\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.981153 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-systemd\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.981153 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h4nl\" (UniqueName: \"kubernetes.io/projected/ab3c3f13-22e3-43af-ad98-06fb882fd52b-kube-api-access-2h4nl\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.981153 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981132    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysctl-d\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysctl-conf\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981178    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-log-socket\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981158    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.981349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-slash\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981337    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n28dm\""
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981365    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-cni-bin\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovnkube-config\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-socket-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-host\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/168e0afb-9a13-4852-81a0-c8d83a365894-etc-tuned\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981480    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168e0afb-9a13-4852-81a0-c8d83a365894-tmp\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d0b83-2a3c-461f-8214-d25f5fbcd63b-tmp-dir\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-ovn\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.981676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6zhb\" (UniqueName: \"kubernetes.io/projected/feb84e54-ff61-45af-9c19-a3b231e5c3e0-kube-api-access-v6zhb\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbxp\" (UniqueName: \"kubernetes.io/projected/182dd4b2-6f32-484f-9fcc-7bd0de43f465-kube-api-access-hpbxp\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysconfig\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-run-netns\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-var-lib-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-registration-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.981995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-modprobe-d\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982063    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-system-cni-dir\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-run\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.982181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-sys\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-kubernetes\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982234    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/671d0b83-2a3c-461f-8214-d25f5fbcd63b-hosts-file\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-sys-fs\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982314    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-var-lib-kubelet\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-kubelet\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982346    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982360    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-systemd-units\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-node-log\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.982761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-cni-netd\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.983137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovn-node-metrics-cert\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.983137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982880    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.983137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982911    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-device-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.983137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.982952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:51.983137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-systemd\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.983399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983147    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovnkube-script-lib\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.983399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdfdbe55-3611-49af-8161-9ba335bbc66f-kubelet-config\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:51.983399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.983399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-lib-modules\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.983399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ppxz\" (UniqueName: \"kubernetes.io/projected/168e0afb-9a13-4852-81a0-c8d83a365894-kube-api-access-6ppxz\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:51.983399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4sj\" (UniqueName: \"kubernetes.io/projected/671d0b83-2a3c-461f-8214-d25f5fbcd63b-kube-api-access-xl4sj\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:51.983664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.983664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-env-overrides\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.983664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tt22\" (UniqueName: \"kubernetes.io/projected/eebd7099-5063-44b0-a6de-ea504ed3b872-kube-api-access-2tt22\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:51.983802 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cnibin\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:51.983850 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983812    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdfdbe55-3611-49af-8161-9ba335bbc66f-dbus\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:51.983911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.983894    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-etc-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:51.984168 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.984007    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 19 08:20:51.984168 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.984118    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g9d4h\""
Sep 19 08:20:51.986091 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.985968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:51.987746 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.987710    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 19 08:20:51.988273 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.988143    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 19 08:20:51.988922 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.988501    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 19 08:20:51.988922 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.988518    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jkt2p\""
Sep 19 08:20:51.988922 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:51.988612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 19 08:20:52.002562 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.002542    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nv7g5"
Sep 19 08:20:52.009838 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.009817    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nv7g5"
Sep 19 08:20:52.070892 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.070865    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 19 08:20:52.077852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.077812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal" event={"ID":"b28b83e27db47df5e96b9958be391357","Type":"ContainerStarted","Data":"60fa7622b9ca2dc57b4ea30d0f5046ed4e2277f828c631af0f99bdd06c87ff32"}
Sep 19 08:20:52.079100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.079075    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal" event={"ID":"78b47eda2825784ab15cc01c09e532f6","Type":"ContainerStarted","Data":"9168cc5ccf49e7178a46c1c4fd5eadaf829f31dc8be353f4d5c7d26d8a609c4a"}
Sep 19 08:20:52.084361 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d022e4-98aa-4524-9f4a-cd7098b35158-cni-binary-copy\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.084475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.084475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.084475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-slash\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-cni-bin\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovnkube-config\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngs6q\" (UniqueName: \"kubernetes.io/projected/56d022e4-98aa-4524-9f4a-cd7098b35158-kube-api-access-ngs6q\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084485    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-socket-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-host\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/168e0afb-9a13-4852-81a0-c8d83a365894-etc-tuned\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168e0afb-9a13-4852-81a0-c8d83a365894-tmp\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d0b83-2a3c-461f-8214-d25f5fbcd63b-tmp-dir\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084576    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-ovn\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-slash\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-host\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-cni-bin\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.084753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-ovn\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-run-ovn-kubernetes\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-socket-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6zhb\" (UniqueName: \"kubernetes.io/projected/feb84e54-ff61-45af-9c19-a3b231e5c3e0-kube-api-access-v6zhb\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/671d0b83-2a3c-461f-8214-d25f5fbcd63b-tmp-dir\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.084999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085051    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbxp\" (UniqueName: \"kubernetes.io/projected/182dd4b2-6f32-484f-9fcc-7bd0de43f465-kube-api-access-hpbxp\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysconfig\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085121    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-cni-multus\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-etc-kubernetes\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/785677a9-31dc-43d7-af71-04178a2ffc21-host-slash\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovnkube-config\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6397ebc5-f249-4d2d-bf1e-183985254161-serviceca\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.085302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-run-netns\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085315    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-run-netns\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysconfig\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-var-lib-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-registration-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085456    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-modprobe-d\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085472    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-var-lib-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-conf-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-system-cni-dir\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085548    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-registration-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-run\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.085565    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-cni-bin\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-system-cni-dir\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.086003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ab3c3f13-22e3-43af-ad98-06fb882fd52b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.085640    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.585620165 +0000 UTC m=+4.771812118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085659    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-daemon-config\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-run\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhnn\" (UniqueName: \"kubernetes.io/projected/785677a9-31dc-43d7-af71-04178a2ffc21-kube-api-access-vwhnn\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-modprobe-d\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-sys\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085757    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085786    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-kubernetes\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/671d0b83-2a3c-461f-8214-d25f5fbcd63b-hosts-file\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085759    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-sys\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-hostroot\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/671d0b83-2a3c-461f-8214-d25f5fbcd63b-hosts-file\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-kubernetes\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-sys-fs\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-var-lib-kubelet\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.086781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-kubelet\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.085985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-systemd-units\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-kubelet\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-var-lib-kubelet\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-systemd-units\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-node-log\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-sys-fs\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-cni-netd\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-node-log\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovn-node-metrics-cert\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-cni-netd\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-socket-dir-parent\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-kubelet\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-multus-certs\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/785677a9-31dc-43d7-af71-04178a2ffc21-iptables-alerter-script\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6397ebc5-f249-4d2d-bf1e-183985254161-host\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.087641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-device-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086342    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-systemd\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovnkube-script-lib\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9614d0-96cb-451d-b53e-c442ef9aebb6-konnectivity-ca\") pod \"konnectivity-agent-m87q2\" (UID: \"1c9614d0-96cb-451d-b53e-c442ef9aebb6\") " pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdfdbe55-3611-49af-8161-9ba335bbc66f-kubelet-config\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.086512    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-device-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-systemd\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.086831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdfdbe55-3611-49af-8161-9ba335bbc66f-kubelet-config\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-lib-modules\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.087084    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.587067354 +0000 UTC m=+4.773259302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ppxz\" (UniqueName: \"kubernetes.io/projected/168e0afb-9a13-4852-81a0-c8d83a365894-kube-api-access-6ppxz\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4sj\" (UniqueName: \"kubernetes.io/projected/671d0b83-2a3c-461f-8214-d25f5fbcd63b-kube-api-access-xl4sj\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.088443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-lib-modules\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovnkube-script-lib\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087183    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-env-overrides\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tt22\" (UniqueName: \"kubernetes.io/projected/eebd7099-5063-44b0-a6de-ea504ed3b872-kube-api-access-2tt22\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cnibin\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-cni-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-cnibin\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-k8s-cni-cncf-io\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087566    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9614d0-96cb-451d-b53e-c442ef9aebb6-agent-certs\") pod \"konnectivity-agent-m87q2\" (UID: \"1c9614d0-96cb-451d-b53e-c442ef9aebb6\") " pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-587jd\" (UniqueName: \"kubernetes.io/projected/6397ebc5-f249-4d2d-bf1e-183985254161-kube-api-access-587jd\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdfdbe55-3611-49af-8161-9ba335bbc66f-dbus\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-etc-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-etc-selinux\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-systemd\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-netns\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.089161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/feb84e54-ff61-45af-9c19-a3b231e5c3e0-env-overrides\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h4nl\" (UniqueName: \"kubernetes.io/projected/ab3c3f13-22e3-43af-ad98-06fb882fd52b-kube-api-access-2h4nl\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysctl-d\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysctl-conf\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-log-socket\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-system-cni-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.087979    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdfdbe55-3611-49af-8161-9ba335bbc66f-dbus\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-etc-openvswitch\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088012    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysctl-conf\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/eebd7099-5063-44b0-a6de-ea504ed3b872-etc-selinux\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-log-socket\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/168e0afb-9a13-4852-81a0-c8d83a365894-etc-sysctl-d\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088166    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab3c3f13-22e3-43af-ad98-06fb882fd52b-cnibin\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.088181    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/feb84e54-ff61-45af-9c19-a3b231e5c3e0-run-systemd\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.090011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.089711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/168e0afb-9a13-4852-81a0-c8d83a365894-etc-tuned\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.090880 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.090828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168e0afb-9a13-4852-81a0-c8d83a365894-tmp\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.091243 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.091172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/feb84e54-ff61-45af-9c19-a3b231e5c3e0-ovn-node-metrics-cert\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.098390 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.097702    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ppxz\" (UniqueName: \"kubernetes.io/projected/168e0afb-9a13-4852-81a0-c8d83a365894-kube-api-access-6ppxz\") pod \"tuned-j8xz4\" (UID: \"168e0afb-9a13-4852-81a0-c8d83a365894\") " pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.098390 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.098087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4sj\" (UniqueName: \"kubernetes.io/projected/671d0b83-2a3c-461f-8214-d25f5fbcd63b-kube-api-access-xl4sj\") pod \"node-resolver-q55gs\" (UID: \"671d0b83-2a3c-461f-8214-d25f5fbcd63b\") " pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.098390 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.098343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6zhb\" (UniqueName: \"kubernetes.io/projected/feb84e54-ff61-45af-9c19-a3b231e5c3e0-kube-api-access-v6zhb\") pod \"ovnkube-node-78n4r\" (UID: \"feb84e54-ff61-45af-9c19-a3b231e5c3e0\") " pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.102057 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.098616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbxp\" (UniqueName: \"kubernetes.io/projected/182dd4b2-6f32-484f-9fcc-7bd0de43f465-kube-api-access-hpbxp\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:52.102344 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.102303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tt22\" (UniqueName: \"kubernetes.io/projected/eebd7099-5063-44b0-a6de-ea504ed3b872-kube-api-access-2tt22\") pod \"aws-ebs-csi-driver-node-9mzz4\" (UID: \"eebd7099-5063-44b0-a6de-ea504ed3b872\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.102694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.102672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h4nl\" (UniqueName: \"kubernetes.io/projected/ab3c3f13-22e3-43af-ad98-06fb882fd52b-kube-api-access-2h4nl\") pod \"multus-additional-cni-plugins-xtf4w\" (UID: \"ab3c3f13-22e3-43af-ad98-06fb882fd52b\") " pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.189203 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-cni-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189203 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-cnibin\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189203 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-k8s-cni-cncf-io\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189203 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9614d0-96cb-451d-b53e-c442ef9aebb6-agent-certs\") pod \"konnectivity-agent-m87q2\" (UID: \"1c9614d0-96cb-451d-b53e-c442ef9aebb6\") " pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-587jd\" (UniqueName: \"kubernetes.io/projected/6397ebc5-f249-4d2d-bf1e-183985254161-kube-api-access-587jd\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-netns\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-k8s-cni-cncf-io\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-cni-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189291    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-system-cni-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-system-cni-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d022e4-98aa-4524-9f4a-cd7098b35158-cni-binary-copy\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngs6q\" (UniqueName: \"kubernetes.io/projected/56d022e4-98aa-4524-9f4a-cd7098b35158-kube-api-access-ngs6q\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-cni-multus\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-etc-kubernetes\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189423    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/785677a9-31dc-43d7-af71-04178a2ffc21-host-slash\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6397ebc5-f249-4d2d-bf1e-183985254161-serviceca\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189483    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-conf-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.189499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-cni-bin\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-cni-multus\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/785677a9-31dc-43d7-af71-04178a2ffc21-host-slash\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189603    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-cni-bin\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189592    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-cnibin\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-daemon-config\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189638    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-etc-kubernetes\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-conf-dir\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhnn\" (UniqueName: \"kubernetes.io/projected/785677a9-31dc-43d7-af71-04178a2ffc21-kube-api-access-vwhnn\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-hostroot\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-netns\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-socket-dir-parent\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189784    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-kubelet\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-multus-certs\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/785677a9-31dc-43d7-af71-04178a2ffc21-iptables-alerter-script\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-hostroot\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6397ebc5-f249-4d2d-bf1e-183985254161-host\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.190120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9614d0-96cb-451d-b53e-c442ef9aebb6-konnectivity-ca\") pod \"konnectivity-agent-m87q2\" (UID: \"1c9614d0-96cb-451d-b53e-c442ef9aebb6\") " pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56d022e4-98aa-4524-9f4a-cd7098b35158-cni-binary-copy\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189941    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-socket-dir-parent\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6397ebc5-f249-4d2d-bf1e-183985254161-serviceca\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-run-multus-certs\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56d022e4-98aa-4524-9f4a-cd7098b35158-host-var-lib-kubelet\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.189999    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6397ebc5-f249-4d2d-bf1e-183985254161-host\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.190214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/56d022e4-98aa-4524-9f4a-cd7098b35158-multus-daemon-config\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.190462    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c9614d0-96cb-451d-b53e-c442ef9aebb6-konnectivity-ca\") pod \"konnectivity-agent-m87q2\" (UID: \"1c9614d0-96cb-451d-b53e-c442ef9aebb6\") " pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.190821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.190556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/785677a9-31dc-43d7-af71-04178a2ffc21-iptables-alerter-script\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.192031 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.192008    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c9614d0-96cb-451d-b53e-c442ef9aebb6-agent-certs\") pod \"konnectivity-agent-m87q2\" (UID: \"1c9614d0-96cb-451d-b53e-c442ef9aebb6\") " pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.196244 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.196210    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:52.196354 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.196251    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:52.196354 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.196264    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.196354 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.196328    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:52.696309935 +0000 UTC m=+4.882501890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.198152 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.198131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngs6q\" (UniqueName: \"kubernetes.io/projected/56d022e4-98aa-4524-9f4a-cd7098b35158-kube-api-access-ngs6q\") pod \"multus-whtmv\" (UID: \"56d022e4-98aa-4524-9f4a-cd7098b35158\") " pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.198406 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.198382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-587jd\" (UniqueName: \"kubernetes.io/projected/6397ebc5-f249-4d2d-bf1e-183985254161-kube-api-access-587jd\") pod \"node-ca-mpps7\" (UID: \"6397ebc5-f249-4d2d-bf1e-183985254161\") " pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.198515 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.198473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhnn\" (UniqueName: \"kubernetes.io/projected/785677a9-31dc-43d7-af71-04178a2ffc21-kube-api-access-vwhnn\") pod \"iptables-alerter-xlzz6\" (UID: \"785677a9-31dc-43d7-af71-04178a2ffc21\") " pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.267965 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.267935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j8xz4"
Sep 19 08:20:52.282207 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.282175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q55gs"
Sep 19 08:20:52.293527 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.293504    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:20:52.300274 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.300257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4"
Sep 19 08:20:52.306739 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.306721    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xtf4w"
Sep 19 08:20:52.314374 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.314355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-whtmv"
Sep 19 08:20:52.321797 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.321778    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xlzz6"
Sep 19 08:20:52.328499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.328302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:20:52.334959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.334943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mpps7"
Sep 19 08:20:52.592876 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.592842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:52.593064 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.592914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:52.593064 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.593023    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.593161 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.593108    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:53.593088012 +0000 UTC m=+5.779279951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:52.593161 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.593025    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.593253 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.593168    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:53.593156339 +0000 UTC m=+5.779348274 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:52.793994 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:52.793945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:52.794166 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.794107    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:52.794166 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.794124    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:52.794166 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.794133    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.794328 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:52.794197    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:53.794179854 +0000 UTC m=+5.980371802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:52.978441 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.978412    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168e0afb_9a13_4852_81a0_c8d83a365894.slice/crio-65612568579c062028e01dc58da9c34cea97cc7a5bb21be1b11136575a614906 WatchSource:0}: Error finding container 65612568579c062028e01dc58da9c34cea97cc7a5bb21be1b11136575a614906: Status 404 returned error can't find the container with id 65612568579c062028e01dc58da9c34cea97cc7a5bb21be1b11136575a614906
Sep 19 08:20:52.979576 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.979552    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3c3f13_22e3_43af_ad98_06fb882fd52b.slice/crio-688fdd532353ef2758ef60e7cf59125fcc3abc6b7e0ca2d186d10089736904ef WatchSource:0}: Error finding container 688fdd532353ef2758ef60e7cf59125fcc3abc6b7e0ca2d186d10089736904ef: Status 404 returned error can't find the container with id 688fdd532353ef2758ef60e7cf59125fcc3abc6b7e0ca2d186d10089736904ef
Sep 19 08:20:52.980325 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.980286    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6397ebc5_f249_4d2d_bf1e_183985254161.slice/crio-e17cd1b8f7fa955732588a579c232a3a271b13d4a20d600a412bb0955506b998 WatchSource:0}: Error finding container e17cd1b8f7fa955732588a579c232a3a271b13d4a20d600a412bb0955506b998: Status 404 returned error can't find the container with id e17cd1b8f7fa955732588a579c232a3a271b13d4a20d600a412bb0955506b998
Sep 19 08:20:52.982753 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.982731    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod671d0b83_2a3c_461f_8214_d25f5fbcd63b.slice/crio-b09a4f76d2f915ab076a80df6e1ffdd859aa2163fd7f1e04c5265ac7822b2a7e WatchSource:0}: Error finding container b09a4f76d2f915ab076a80df6e1ffdd859aa2163fd7f1e04c5265ac7822b2a7e: Status 404 returned error can't find the container with id b09a4f76d2f915ab076a80df6e1ffdd859aa2163fd7f1e04c5265ac7822b2a7e
Sep 19 08:20:52.984880 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.984851    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785677a9_31dc_43d7_af71_04178a2ffc21.slice/crio-79636aafeadeae29f6e0e8655bec46febb6e2e6fad1250f74ef1275299f0a5bf WatchSource:0}: Error finding container 79636aafeadeae29f6e0e8655bec46febb6e2e6fad1250f74ef1275299f0a5bf: Status 404 returned error can't find the container with id 79636aafeadeae29f6e0e8655bec46febb6e2e6fad1250f74ef1275299f0a5bf
Sep 19 08:20:52.985918 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.985895    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9614d0_96cb_451d_b53e_c442ef9aebb6.slice/crio-dd3e082785e1a0ec05870ad6de9348df9d1e29a2298ed843fa66b3f04975e779 WatchSource:0}: Error finding container dd3e082785e1a0ec05870ad6de9348df9d1e29a2298ed843fa66b3f04975e779: Status 404 returned error can't find the container with id dd3e082785e1a0ec05870ad6de9348df9d1e29a2298ed843fa66b3f04975e779
Sep 19 08:20:52.987092 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.987071    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d022e4_98aa_4524_9f4a_cd7098b35158.slice/crio-2e26457003305e409ae11b1e81f7f4486cfce3ce4c7bf64425b8f8c1b06bc8aa WatchSource:0}: Error finding container 2e26457003305e409ae11b1e81f7f4486cfce3ce4c7bf64425b8f8c1b06bc8aa: Status 404 returned error can't find the container with id 2e26457003305e409ae11b1e81f7f4486cfce3ce4c7bf64425b8f8c1b06bc8aa
Sep 19 08:20:52.988342 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:20:52.988300    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb84e54_ff61_45af_9c19_a3b231e5c3e0.slice/crio-939afdf1570522a79b256e177a8220b9ae92f42ff8a964cf142f2c0217a861ee WatchSource:0}: Error finding container 939afdf1570522a79b256e177a8220b9ae92f42ff8a964cf142f2c0217a861ee: Status 404 returned error can't find the container with id 939afdf1570522a79b256e177a8220b9ae92f42ff8a964cf142f2c0217a861ee
Sep 19 08:20:53.013933 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.013884    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:52 +0000 UTC" deadline="2028-07-02 23:48:22.489586882 +0000 UTC"
Sep 19 08:20:53.013933 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.013932    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15663h27m29.475658578s"
Sep 19 08:20:53.073664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.073641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:53.073796 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.073777    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:20:53.081912 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.081886    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"939afdf1570522a79b256e177a8220b9ae92f42ff8a964cf142f2c0217a861ee"}
Sep 19 08:20:53.083202 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.083177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-whtmv" event={"ID":"56d022e4-98aa-4524-9f4a-cd7098b35158","Type":"ContainerStarted","Data":"2e26457003305e409ae11b1e81f7f4486cfce3ce4c7bf64425b8f8c1b06bc8aa"}
Sep 19 08:20:53.084372 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.084350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mpps7" event={"ID":"6397ebc5-f249-4d2d-bf1e-183985254161","Type":"ContainerStarted","Data":"e17cd1b8f7fa955732588a579c232a3a271b13d4a20d600a412bb0955506b998"}
Sep 19 08:20:53.085578 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.085542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4" event={"ID":"eebd7099-5063-44b0-a6de-ea504ed3b872","Type":"ContainerStarted","Data":"7f981fb513863ccd50e5386fbdbed04f94ca0b3c005ce4d4b1d2187f624d2b0a"}
Sep 19 08:20:53.086699 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.086673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m87q2" event={"ID":"1c9614d0-96cb-451d-b53e-c442ef9aebb6","Type":"ContainerStarted","Data":"dd3e082785e1a0ec05870ad6de9348df9d1e29a2298ed843fa66b3f04975e779"}
Sep 19 08:20:53.087853 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.087836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlzz6" event={"ID":"785677a9-31dc-43d7-af71-04178a2ffc21","Type":"ContainerStarted","Data":"79636aafeadeae29f6e0e8655bec46febb6e2e6fad1250f74ef1275299f0a5bf"}
Sep 19 08:20:53.088881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.088853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q55gs" event={"ID":"671d0b83-2a3c-461f-8214-d25f5fbcd63b","Type":"ContainerStarted","Data":"b09a4f76d2f915ab076a80df6e1ffdd859aa2163fd7f1e04c5265ac7822b2a7e"}
Sep 19 08:20:53.090131 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.090108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerStarted","Data":"688fdd532353ef2758ef60e7cf59125fcc3abc6b7e0ca2d186d10089736904ef"}
Sep 19 08:20:53.091318 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.091291    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j8xz4" event={"ID":"168e0afb-9a13-4852-81a0-c8d83a365894","Type":"ContainerStarted","Data":"65612568579c062028e01dc58da9c34cea97cc7a5bb21be1b11136575a614906"}
Sep 19 08:20:53.093296 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.093274    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal" event={"ID":"b28b83e27db47df5e96b9958be391357","Type":"ContainerStarted","Data":"66f4e7f5e377615e4fff74bc72faa2f3a889252248b7a3ed2c937b325f570b34"}
Sep 19 08:20:53.110723 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.110683    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-191.ec2.internal" podStartSLOduration=2.110669856 podStartE2EDuration="2.110669856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:53.110369774 +0000 UTC m=+5.296561727" watchObservedRunningTime="2026-09-19 08:20:53.110669856 +0000 UTC m=+5.296861803"
Sep 19 08:20:53.602015 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.601256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:53.602015 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.601379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:53.602015 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.601527    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:53.602015 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.601590    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:55.601570717 +0000 UTC m=+7.787762666 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:53.602015 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.601773    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:53.602015 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.601820    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:55.601806343 +0000 UTC m=+7.787998284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:53.803141 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:53.803100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:53.803337 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.803262    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:53.803337 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.803282    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:53.803337 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.803294    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:53.803486 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:53.803350    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:55.803332085 +0000 UTC m=+7.989524039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:54.014803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:54.014715    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:52 +0000 UTC" deadline="2028-06-12 02:57:04.98600711 +0000 UTC"
Sep 19 08:20:54.014803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:54.014777    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15162h36m10.971233505s"
Sep 19 08:20:54.076388 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:54.076354    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:54.076556 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:54.076475    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:20:54.077370 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:54.076851    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:54.077370 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:54.076954    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:20:54.108471 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:54.108134    2580 generic.go:356] "Generic (PLEG): container finished" podID="78b47eda2825784ab15cc01c09e532f6" containerID="502ff4359b58a6e0930d229c2ea1c54fa8912e2d40aeed0188cbc5da81e5e518" exitCode=0
Sep 19 08:20:54.108719 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:54.108660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal" event={"ID":"78b47eda2825784ab15cc01c09e532f6","Type":"ContainerDied","Data":"502ff4359b58a6e0930d229c2ea1c54fa8912e2d40aeed0188cbc5da81e5e518"}
Sep 19 08:20:55.076818 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:55.076302    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:55.076818 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.076460    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:20:55.136678 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:55.136009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal" event={"ID":"78b47eda2825784ab15cc01c09e532f6","Type":"ContainerStarted","Data":"5db08a28d3a519e545edbdb5aa05501176437705b42de10d7e12c2e258e22c96"}
Sep 19 08:20:55.208570 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:55.208506    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-191.ec2.internal" podStartSLOduration=4.208483797 podStartE2EDuration="4.208483797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.207251126 +0000 UTC m=+7.393443081" watchObservedRunningTime="2026-09-19 08:20:55.208483797 +0000 UTC m=+7.394675755"
Sep 19 08:20:55.621440 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:55.621404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:55.621635 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:55.621479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:55.621635 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.621614    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:55.621745 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.621672    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:59.621655067 +0000 UTC m=+11.807847006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:55.622064 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.622040    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:55.622148 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.622124    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:20:59.622109216 +0000 UTC m=+11.808301157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:55.822628 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:55.822592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:55.822809 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.822746    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:55.822809 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.822764    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:55.822809 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.822776    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:55.822966 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:55.822833    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:59.822815266 +0000 UTC m=+12.009007204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:56.077151 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:56.076425    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:56.077151 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:56.076554    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:20:56.077151 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:56.076616    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:56.077151 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:56.076685    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:20:57.073873 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:57.073841    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:57.074060 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:57.073985    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:20:58.076021 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:58.075988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:58.076472 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:58.076101    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:20:58.076472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:58.076313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:58.076472 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:58.076398    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:20:59.074043 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:59.074008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:59.074244 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.074160    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:20:59.656112 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:59.656032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:20:59.656612 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:59.656171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:20:59.656612 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.656329    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:59.656612 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.656395    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.656376665 +0000 UTC m=+19.842568600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:20:59.656612 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.656329    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:59.656612 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.656517    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.656493499 +0000 UTC m=+19.842685448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:20:59.858135 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:20:59.858084    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:20:59.858344 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.858264    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:20:59.858344 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.858284    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:20:59.858344 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.858293    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:20:59.858344 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:20:59.858345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.85833249 +0000 UTC m=+20.044524425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:00.074918 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:00.074880    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:00.074918 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:00.074921    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:00.075135 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:00.075011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:00.075135 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:00.075107    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:01.074915 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:01.074337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:01.074915 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:01.074707    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:02.074572 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:02.074542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:02.074761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:02.074541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:02.074761 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:02.074672    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:02.074876 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:02.074805    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:03.074072 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:03.074019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:03.074564 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:03.074187    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:04.073747 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:04.073713    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:04.074004 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:04.073761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:04.074004 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:04.073854    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:04.074004 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:04.073980    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:05.073676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:05.073646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:05.074121 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:05.073776    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:06.073514 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:06.073482    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:06.073672 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:06.073623    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:06.073672 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:06.073658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:06.074024 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:06.073742    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:07.074496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:07.074460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:07.075005 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.074602    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:07.711107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:07.711070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:07.711302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:07.711140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:07.711302 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.711240    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:07.711302 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.711264    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:07.711458 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.711306    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.711286387 +0000 UTC m=+35.897478323 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:07.711458 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.711330    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.71131823 +0000 UTC m=+35.897510165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:07.912671 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:07.912635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:07.912833 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.912805    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:07.912833 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.912823    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:07.912956 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.912837    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:07.912956 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:07.912891    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:23.912872182 +0000 UTC m=+36.099064139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:08.074569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:08.074485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:08.074569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:08.074529    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:08.075053 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:08.074628    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:08.075053 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:08.074759    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:09.073926 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:09.073888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:09.074122 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:09.074025    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:10.074632 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:10.074599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:10.074997 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:10.074711    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:10.074997 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:10.074746    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:10.074997 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:10.074813    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:11.073902 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.073722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:11.074061 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:11.074004    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:11.170577 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.170502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m87q2" event={"ID":"1c9614d0-96cb-451d-b53e-c442ef9aebb6","Type":"ContainerStarted","Data":"cf4ea5b7ec9896d52e734246a8637150979f953a264ce58e8c6806a1339808bd"}
Sep 19 08:21:11.172288 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.172258    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q55gs" event={"ID":"671d0b83-2a3c-461f-8214-d25f5fbcd63b","Type":"ContainerStarted","Data":"f32d483b5839dc4aa48d6b0525acbe78a75a5a7514076a02857ebfa06a7aeee5"}
Sep 19 08:21:11.174724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.174466    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab3c3f13-22e3-43af-ad98-06fb882fd52b" containerID="f0c4e6303d460a9e9d34c31c8f4be2937a6fc85820c71ab76b8e1b2f0a4e5a73" exitCode=0
Sep 19 08:21:11.174724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.174534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerDied","Data":"f0c4e6303d460a9e9d34c31c8f4be2937a6fc85820c71ab76b8e1b2f0a4e5a73"}
Sep 19 08:21:11.177200 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.177175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j8xz4" event={"ID":"168e0afb-9a13-4852-81a0-c8d83a365894","Type":"ContainerStarted","Data":"0fd8ad94a86a4bb5433e1e441b8c26d3685035a1e5c68e84e0144288be225d11"}
Sep 19 08:21:11.180742 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.180712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"a1a61e109c76d44cb92724f8679cc008df30e4f566b8a23bc76d3bd7804d28ec"}
Sep 19 08:21:11.180845 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.180784    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"607ebbc329114d9e7bb37fa3b095266c8e83d9cd94637b1269d1ec5bbc9f620e"}
Sep 19 08:21:11.180845 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.180820    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"e9ed12e356b2ae0ad10ad7e5504d5c55af98d2bfc3e2e46bcf74a0fbe574d930"}
Sep 19 08:21:11.180978 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.180847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"d3f828a5b58535cf38605c7505b4161b5f4cf6beb2a9faa4cf592984b826b605"}
Sep 19 08:21:11.180978 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.180884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"de98c3bb6cdb0b2a72da230ccc02afdb97afa34cd7b35188a0fceaed7d8a7683"}
Sep 19 08:21:11.180978 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.180919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"ce3bda1bb57ef8b1b1d4b21431fa04c4f8f4ec0c24992f06034e0378c4774e86"}
Sep 19 08:21:11.182270 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.182243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-whtmv" event={"ID":"56d022e4-98aa-4524-9f4a-cd7098b35158","Type":"ContainerStarted","Data":"6e2568888a278336d70b00104dffcb78bd7a04c248673b2f80af377283c0bc1f"}
Sep 19 08:21:11.183817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.183794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mpps7" event={"ID":"6397ebc5-f249-4d2d-bf1e-183985254161","Type":"ContainerStarted","Data":"73049e08b917c19384d656cd51e30da7b206577febf9ac8409b17eece76d0a07"}
Sep 19 08:21:11.185765 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.185745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4" event={"ID":"eebd7099-5063-44b0-a6de-ea504ed3b872","Type":"ContainerStarted","Data":"c2ff48f6886c90bc69dc8e57581fa5064bd0dbf72ca12156fb636af4845f476c"}
Sep 19 08:21:11.186090 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.186040    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m87q2" podStartSLOduration=11.282418273 podStartE2EDuration="21.186023592s" totalImagesPullingTime="9.903605319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.185699269 +0000 UTC m=+23.371891238" watchObservedRunningTime="2026-09-19 08:21:11.186023592 +0000 UTC m=+23.372215551"
Sep 19 08:21:11.242245 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.242185    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-whtmv" podStartSLOduration=3.818784655 podStartE2EDuration="21.24216507s" totalImagesPullingTime="17.423380415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.226891568 +0000 UTC m=+23.413083526" watchObservedRunningTime="2026-09-19 08:21:11.24216507 +0000 UTC m=+23.428357027"
Sep 19 08:21:11.243000 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.242959    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mpps7" podStartSLOduration=3.85499823 podStartE2EDuration="21.242949711s" totalImagesPullingTime="17.387951481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.242040103 +0000 UTC m=+23.428232063" watchObservedRunningTime="2026-09-19 08:21:11.242949711 +0000 UTC m=+23.429141670"
Sep 19 08:21:11.264153 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.264119    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j8xz4" podStartSLOduration=3.872431442 podStartE2EDuration="21.264106495s" totalImagesPullingTime="17.391675053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.263868768 +0000 UTC m=+23.450060724" watchObservedRunningTime="2026-09-19 08:21:11.264106495 +0000 UTC m=+23.450298452"
Sep 19 08:21:11.279147 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.279110    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q55gs" podStartSLOduration=3.922611827 podStartE2EDuration="21.279097679s" totalImagesPullingTime="17.356485852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:11.278735306 +0000 UTC m=+23.464927262" watchObservedRunningTime="2026-09-19 08:21:11.279097679 +0000 UTC m=+23.465289636"
Sep 19 08:21:11.394100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:11.393889    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 19 08:21:12.059298 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.059176    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:21:11.394097972Z","UUID":"5071c14d-159a-4991-bad4-d693a4269b9f","Handler":null,"Name":"","Endpoint":""}
Sep 19 08:21:12.060982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.060951    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 19 08:21:12.061109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.060992    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 19 08:21:12.074145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.074119    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:12.074299 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:12.074251    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:12.074623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.074589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:12.074734 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:12.074688    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:12.190295 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.190261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4" event={"ID":"eebd7099-5063-44b0-a6de-ea504ed3b872","Type":"ContainerStarted","Data":"76384d8f3d31d7d81899db7877803cfe01b34ea4229369affa923336a01bea82"}
Sep 19 08:21:12.192661 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.192634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xlzz6" event={"ID":"785677a9-31dc-43d7-af71-04178a2ffc21","Type":"ContainerStarted","Data":"9cf12caefa1bdb327920cfd9df62cd0cec6f9b337f33a9c2fd4a7b5cb59f9e06"}
Sep 19 08:21:12.208871 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:12.208822    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xlzz6" podStartSLOduration=4.8542354549999995 podStartE2EDuration="22.208803771s" totalImagesPullingTime="17.354568316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:12.208571034 +0000 UTC m=+24.394762992" watchObservedRunningTime="2026-09-19 08:21:12.208803771 +0000 UTC m=+24.394995731"
Sep 19 08:21:13.074327 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:13.074202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:13.074486 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:13.074439    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:13.137172 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:13.137138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:21:13.137736 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:13.137714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:21:13.199602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:13.198532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4" event={"ID":"eebd7099-5063-44b0-a6de-ea504ed3b872","Type":"ContainerStarted","Data":"e696bea7d41821a4d64917cddf808087571dde96b284347b5cef5ec70c605845"}
Sep 19 08:21:13.203132 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:13.203091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"67707f0a15c1ebad5c79c31f8a1f85928da220dd2c763caa92e6d4e9de63fc37"}
Sep 19 08:21:13.216360 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:13.216317    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9mzz4" podStartSLOduration=4.423057573 podStartE2EDuration="23.216303349s" totalImagesPullingTime="18.793245776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:13.216197563 +0000 UTC m=+25.402389521" watchObservedRunningTime="2026-09-19 08:21:13.216303349 +0000 UTC m=+25.402495305"
Sep 19 08:21:14.073803 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:14.073764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:14.074005 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:14.073813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:14.074005 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:14.073895    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:14.074146 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:14.074028    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:14.205916 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:14.205883    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:15.073516 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.073495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:15.073618 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:15.073601    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:15.213315 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.213130    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" event={"ID":"feb84e54-ff61-45af-9c19-a3b231e5c3e0","Type":"ContainerStarted","Data":"ad35f36fdf032d32432d4a8aaf372598703592749fab3615fb496bd3af8b17e7"}
Sep 19 08:21:15.213623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.213415    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:21:15.213623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.213474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:21:15.213623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.213521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:21:15.234266 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.231462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:21:15.235074 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.235049    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:21:15.242712 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:15.242656    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r" podStartSLOduration=7.849702772 podStartE2EDuration="25.242643088s" totalImagesPullingTime="17.392940316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:15.242320416 +0000 UTC m=+27.428512374" watchObservedRunningTime="2026-09-19 08:21:15.242643088 +0000 UTC m=+27.428835073"
Sep 19 08:21:16.074314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:16.074282    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:16.074314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:16.074314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:16.074507 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:16.074390    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:16.074507 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:16.074493    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:16.216674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:16.216642    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab3c3f13-22e3-43af-ad98-06fb882fd52b" containerID="d4ce6a4b65574285786c5c355cf28c10ed88996e1e44e7577a691aea9795d4a9" exitCode=0
Sep 19 08:21:16.217418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:16.216723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerDied","Data":"d4ce6a4b65574285786c5c355cf28c10ed88996e1e44e7577a691aea9795d4a9"}
Sep 19 08:21:17.073892 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.073860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:17.074059 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:17.073965    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:17.150398 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.150368    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ttzbh"]
Sep 19 08:21:17.150541 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.150460    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:17.150581 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:17.150536    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:17.155133 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.155107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jnc2h"]
Sep 19 08:21:17.155343 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.155202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:17.155343 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:17.155318    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:17.155810 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.155786    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkxrb"]
Sep 19 08:21:17.219166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.219138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:17.219702 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:17.219281    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:17.512898 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.512647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:21:17.513041 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.512979    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 19 08:21:17.513421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:17.513400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m87q2"
Sep 19 08:21:18.223205 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:18.223172    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab3c3f13-22e3-43af-ad98-06fb882fd52b" containerID="770c2d642c68e70c06b3726ac35520795a11a8d5d874b049c7823809ea246384" exitCode=0
Sep 19 08:21:18.223598 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:18.223265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerDied","Data":"770c2d642c68e70c06b3726ac35520795a11a8d5d874b049c7823809ea246384"}
Sep 19 08:21:19.073868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:19.073839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:19.074047 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:19.073839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:19.074047 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:19.073972    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:19.074047 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:19.073839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:19.074173 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:19.074050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:19.074173 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:19.074096    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:20.231679 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:20.231642    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab3c3f13-22e3-43af-ad98-06fb882fd52b" containerID="8d783999a8ba7584306f6b9f785161b6500b1d7f20266d1366bf0a103ad3fc88" exitCode=0
Sep 19 08:21:20.232060 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:20.231696    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerDied","Data":"8d783999a8ba7584306f6b9f785161b6500b1d7f20266d1366bf0a103ad3fc88"}
Sep 19 08:21:21.073579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:21.073544    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:21.073745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:21.073581    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:21.073745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:21.073584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:21.073745 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:21.073660    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:21.073895 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:21.073733    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:21.073895 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:21.073848    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:23.073714 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.073680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:23.074207 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.073691    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:23.074207 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.073820    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkxrb" podUID="182dd4b2-6f32-484f-9fcc-7bd0de43f465"
Sep 19 08:21:23.074207 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.073864    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:23.074207 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.073942    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ttzbh" podUID="bdfdbe55-3611-49af-8161-9ba335bbc66f"
Sep 19 08:21:23.074207 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.074016    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jnc2h" podUID="8bd653aa-de69-413f-874c-32930fef8b85"
Sep 19 08:21:23.249170 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.249102    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-191.ec2.internal" event="NodeReady"
Sep 19 08:21:23.249348 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.249236    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 19 08:21:23.297753 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.297719    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wqgl6"]
Sep 19 08:21:23.315771 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.315742    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4xst4"]
Sep 19 08:21:23.315932 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.315884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.318970 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.318490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 19 08:21:23.318970 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.318718    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vj75p\""
Sep 19 08:21:23.318970 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.318901    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 19 08:21:23.319235 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.319014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 19 08:21:23.329827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.329798    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqgl6"]
Sep 19 08:21:23.329827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.329831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4xst4"]
Sep 19 08:21:23.330007 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.329925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.330007 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.329938    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jzt4t"]
Sep 19 08:21:23.332462 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.332441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 19 08:21:23.333109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.332741    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qqzk5\""
Sep 19 08:21:23.333109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.332840    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 19 08:21:23.333109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.332869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 19 08:21:23.333109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.332974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 19 08:21:23.348138 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.348103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzt4t"]
Sep 19 08:21:23.348314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.348182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.350663 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.350643    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 19 08:21:23.351158 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.351138    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wm7fh\""
Sep 19 08:21:23.351362 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.351344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 19 08:21:23.421589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msddb\" (UniqueName: \"kubernetes.io/projected/74f1437b-61c4-4a11-bbea-988d75a1d579-kube-api-access-msddb\") pod \"ingress-canary-wqgl6\" (UID: \"74f1437b-61c4-4a11-bbea-988d75a1d579\") " pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.421589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421594    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rttdr\" (UniqueName: \"kubernetes.io/projected/19b17a34-96c9-4497-82d3-e359db7fcf43-kube-api-access-rttdr\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.421841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnf9d\" (UniqueName: \"kubernetes.io/projected/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-api-access-fnf9d\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.421841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/195ae344-bf5f-4194-9dd0-d36dc7e712ad-data-volume\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.421841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.421841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421787    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.421841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19b17a34-96c9-4497-82d3-e359db7fcf43-config-volume\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.422079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19b17a34-96c9-4497-82d3-e359db7fcf43-tmp-dir\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.422079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f1437b-61c4-4a11-bbea-988d75a1d579-cert\") pod \"ingress-canary-wqgl6\" (UID: \"74f1437b-61c4-4a11-bbea-988d75a1d579\") " pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.422079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/195ae344-bf5f-4194-9dd0-d36dc7e712ad-crio-socket\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.422079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.421989    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.522990 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.522905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/195ae344-bf5f-4194-9dd0-d36dc7e712ad-data-volume\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.522990 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.522949    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.522990 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.522988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19b17a34-96c9-4497-82d3-e359db7fcf43-config-volume\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19b17a34-96c9-4497-82d3-e359db7fcf43-tmp-dir\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f1437b-61c4-4a11-bbea-988d75a1d579-cert\") pod \"ingress-canary-wqgl6\" (UID: \"74f1437b-61c4-4a11-bbea-988d75a1d579\") " pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.523139    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.523153    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.523233    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-rbac-proxy-cm podName:195ae344-bf5f-4194-9dd0-d36dc7e712ad nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.023200394 +0000 UTC m=+36.209392344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4xst4" (UID: "195ae344-bf5f-4194-9dd0-d36dc7e712ad") : configmap "kube-rbac-proxy" not found
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.523253    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls podName:195ae344-bf5f-4194-9dd0-d36dc7e712ad nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.023243891 +0000 UTC m=+36.209435827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xst4" (UID: "195ae344-bf5f-4194-9dd0-d36dc7e712ad") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/195ae344-bf5f-4194-9dd0-d36dc7e712ad-crio-socket\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.523342 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523340    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/195ae344-bf5f-4194-9dd0-d36dc7e712ad-data-volume\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msddb\" (UniqueName: \"kubernetes.io/projected/74f1437b-61c4-4a11-bbea-988d75a1d579-kube-api-access-msddb\") pod \"ingress-canary-wqgl6\" (UID: \"74f1437b-61c4-4a11-bbea-988d75a1d579\") " pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523388    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rttdr\" (UniqueName: \"kubernetes.io/projected/19b17a34-96c9-4497-82d3-e359db7fcf43-kube-api-access-rttdr\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523410    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19b17a34-96c9-4497-82d3-e359db7fcf43-tmp-dir\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523419    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnf9d\" (UniqueName: \"kubernetes.io/projected/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-api-access-fnf9d\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/195ae344-bf5f-4194-9dd0-d36dc7e712ad-crio-socket\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.523603    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.523660    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls podName:19b17a34-96c9-4497-82d3-e359db7fcf43 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:24.023641865 +0000 UTC m=+36.209833838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls") pod "dns-default-jzt4t" (UID: "19b17a34-96c9-4497-82d3-e359db7fcf43") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:23.523875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.523715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/19b17a34-96c9-4497-82d3-e359db7fcf43-config-volume\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.528452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.528432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74f1437b-61c4-4a11-bbea-988d75a1d579-cert\") pod \"ingress-canary-wqgl6\" (UID: \"74f1437b-61c4-4a11-bbea-988d75a1d579\") " pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.537348 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.537327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnf9d\" (UniqueName: \"kubernetes.io/projected/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-api-access-fnf9d\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:23.537480 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.537457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rttdr\" (UniqueName: \"kubernetes.io/projected/19b17a34-96c9-4497-82d3-e359db7fcf43-kube-api-access-rttdr\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:23.539151 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.539130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msddb\" (UniqueName: \"kubernetes.io/projected/74f1437b-61c4-4a11-bbea-988d75a1d579-kube-api-access-msddb\") pod \"ingress-canary-wqgl6\" (UID: \"74f1437b-61c4-4a11-bbea-988d75a1d579\") " pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.628237 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.628185    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqgl6"
Sep 19 08:21:23.724718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.724679    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:23.724896 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.724758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:23.724896 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.724845    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:23.725002 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.724857    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:23.725002 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.724932    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret podName:bdfdbe55-3611-49af-8161-9ba335bbc66f nodeName:}" failed. No retries permitted until 2026-09-19 08:21:55.724908125 +0000 UTC m=+67.911100062 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret") pod "global-pull-secret-syncer-ttzbh" (UID: "bdfdbe55-3611-49af-8161-9ba335bbc66f") : object "kube-system"/"original-pull-secret" not registered
Sep 19 08:21:23.725002 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.724980    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs podName:182dd4b2-6f32-484f-9fcc-7bd0de43f465 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:55.724963436 +0000 UTC m=+67.911155371 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs") pod "network-metrics-daemon-dkxrb" (UID: "182dd4b2-6f32-484f-9fcc-7bd0de43f465") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 19 08:21:23.926247 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:23.926144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:23.926419 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.926329    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 19 08:21:23.926419 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.926354    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 19 08:21:23.926419 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.926366    2580 projected.go:196] Error preparing data for projected volume kube-api-access-wbzlw for pod openshift-network-diagnostics/network-check-target-jnc2h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:23.926563 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:23.926428    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw podName:8bd653aa-de69-413f-874c-32930fef8b85 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:55.926411066 +0000 UTC m=+68.112603003 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wbzlw" (UniqueName: "kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw") pod "network-check-target-jnc2h" (UID: "8bd653aa-de69-413f-874c-32930fef8b85") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 19 08:21:24.026993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:24.026768    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:24.027251 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:24.027033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:24.027251 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:24.026957    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:24.027251 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:24.027118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:24.027251 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:24.027143    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls podName:195ae344-bf5f-4194-9dd0-d36dc7e712ad nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.027122639 +0000 UTC m=+37.213314574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xst4" (UID: "195ae344-bf5f-4194-9dd0-d36dc7e712ad") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:24.027251 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:24.027204    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:24.027477 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:24.027282    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls podName:19b17a34-96c9-4497-82d3-e359db7fcf43 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:25.027264985 +0000 UTC m=+37.213456925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls") pod "dns-default-jzt4t" (UID: "19b17a34-96c9-4497-82d3-e359db7fcf43") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:24.027731 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:24.027709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/195ae344-bf5f-4194-9dd0-d36dc7e712ad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:24.532579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:24.532508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqgl6"]
Sep 19 08:21:24.535804 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:21:24.535777    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f1437b_61c4_4a11_bbea_988d75a1d579.slice/crio-90425652f4ef7d9e82ead283aadcc1d01884a952cf7acd447ca1669f883ab007 WatchSource:0}: Error finding container 90425652f4ef7d9e82ead283aadcc1d01884a952cf7acd447ca1669f883ab007: Status 404 returned error can't find the container with id 90425652f4ef7d9e82ead283aadcc1d01884a952cf7acd447ca1669f883ab007
Sep 19 08:21:25.034830 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.034789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:25.035066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.034862    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:25.035066 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:25.034947    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.035066 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:25.035009    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:25.035066 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:25.035025    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls podName:19b17a34-96c9-4497-82d3-e359db7fcf43 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:27.035004638 +0000 UTC m=+39.221196576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls") pod "dns-default-jzt4t" (UID: "19b17a34-96c9-4497-82d3-e359db7fcf43") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:25.035066 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:25.035062    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls podName:195ae344-bf5f-4194-9dd0-d36dc7e712ad nodeName:}" failed. No retries permitted until 2026-09-19 08:21:27.035046014 +0000 UTC m=+39.221237950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xst4" (UID: "195ae344-bf5f-4194-9dd0-d36dc7e712ad") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:25.073668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.073637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:25.073668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.073658    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:25.073894 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.073637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:25.076412 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.076392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:25.077206 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.077185    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:25.077374 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.077355    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jjfx\""
Sep 19 08:21:25.077374 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.077357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vlr4t\""
Sep 19 08:21:25.077508 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.077424    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:25.077508 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.077442    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:25.247609 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.247567    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab3c3f13-22e3-43af-ad98-06fb882fd52b" containerID="0df53034a77dddbfd2cbc87cc5b86f9f20a2978394cca3fa571c22051aee2742" exitCode=0
Sep 19 08:21:25.247816 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.247650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerDied","Data":"0df53034a77dddbfd2cbc87cc5b86f9f20a2978394cca3fa571c22051aee2742"}
Sep 19 08:21:25.249066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.249041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqgl6" event={"ID":"74f1437b-61c4-4a11-bbea-988d75a1d579","Type":"ContainerStarted","Data":"90425652f4ef7d9e82ead283aadcc1d01884a952cf7acd447ca1669f883ab007"}
Sep 19 08:21:25.448639 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:25.448613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q55gs_671d0b83-2a3c-461f-8214-d25f5fbcd63b/dns-node-resolver/0.log"
Sep 19 08:21:26.255355 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:26.255319    2580 generic.go:356] "Generic (PLEG): container finished" podID="ab3c3f13-22e3-43af-ad98-06fb882fd52b" containerID="351ba398fb526d0ffdce1a3b2dda216935052a700cea38a8a2435de068dc0a44" exitCode=0
Sep 19 08:21:26.256100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:26.255379    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerDied","Data":"351ba398fb526d0ffdce1a3b2dda216935052a700cea38a8a2435de068dc0a44"}
Sep 19 08:21:26.418623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:26.418599    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mpps7_6397ebc5-f249-4d2d-bf1e-183985254161/node-ca/0.log"
Sep 19 08:21:27.054444 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:27.054283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:27.054544 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:27.054441    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 19 08:21:27.054544 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:27.054474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:27.054544 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:27.054496    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls podName:19b17a34-96c9-4497-82d3-e359db7fcf43 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:31.054481276 +0000 UTC m=+43.240673246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls") pod "dns-default-jzt4t" (UID: "19b17a34-96c9-4497-82d3-e359db7fcf43") : secret "dns-default-metrics-tls" not found
Sep 19 08:21:27.054694 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:27.054552    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:27.054694 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:27.054601    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls podName:195ae344-bf5f-4194-9dd0-d36dc7e712ad nodeName:}" failed. No retries permitted until 2026-09-19 08:21:31.054587473 +0000 UTC m=+43.240779422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xst4" (UID: "195ae344-bf5f-4194-9dd0-d36dc7e712ad") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:27.259448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:27.259322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqgl6" event={"ID":"74f1437b-61c4-4a11-bbea-988d75a1d579","Type":"ContainerStarted","Data":"d27a82ad832a2a3bc993032fcf96a56a8b1943f7caeac721f8ae9e21b3c3d7f5"}
Sep 19 08:21:27.262744 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:27.262717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" event={"ID":"ab3c3f13-22e3-43af-ad98-06fb882fd52b","Type":"ContainerStarted","Data":"7535510ada7205d5984d3f5ecfd249ffb359735434bbde8d796ffbecc233a90a"}
Sep 19 08:21:27.278780 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:27.278729    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wqgl6" podStartSLOduration=1.87482321 podStartE2EDuration="4.278714195s" totalImagesPullingTime="2.403890985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:27.277730949 +0000 UTC m=+39.463922905" watchObservedRunningTime="2026-09-19 08:21:27.278714195 +0000 UTC m=+39.464906152"
Sep 19 08:21:27.305964 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:27.305912    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xtf4w" podStartSLOduration=9.494230638 podStartE2EDuration="37.305894953s" totalImagesPullingTime="27.811664315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:27.305669078 +0000 UTC m=+39.491861035" watchObservedRunningTime="2026-09-19 08:21:27.305894953 +0000 UTC m=+39.492086912"
Sep 19 08:21:31.084114 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:31.084069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:31.084576 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:31.084141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:31.084576 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:31.084211    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:31.084576 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:31.084293    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls podName:195ae344-bf5f-4194-9dd0-d36dc7e712ad nodeName:}" failed. No retries permitted until 2026-09-19 08:21:39.084277512 +0000 UTC m=+51.270469448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4xst4" (UID: "195ae344-bf5f-4194-9dd0-d36dc7e712ad") : secret "insights-runtime-extractor-tls" not found
Sep 19 08:21:31.087369 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:31.087343    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/19b17a34-96c9-4497-82d3-e359db7fcf43-metrics-tls\") pod \"dns-default-jzt4t\" (UID: \"19b17a34-96c9-4497-82d3-e359db7fcf43\") " pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:31.157843 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:31.157811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:31.425069 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:31.425000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jzt4t"]
Sep 19 08:21:31.428126 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:21:31.428092    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b17a34_96c9_4497_82d3_e359db7fcf43.slice/crio-ae5407341dc23f0beb90d9f1e6c9ff38fdcc6867c41037972834dfb4b67884c4 WatchSource:0}: Error finding container ae5407341dc23f0beb90d9f1e6c9ff38fdcc6867c41037972834dfb4b67884c4: Status 404 returned error can't find the container with id ae5407341dc23f0beb90d9f1e6c9ff38fdcc6867c41037972834dfb4b67884c4
Sep 19 08:21:32.276299 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:32.276256    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzt4t" event={"ID":"19b17a34-96c9-4497-82d3-e359db7fcf43","Type":"ContainerStarted","Data":"ae5407341dc23f0beb90d9f1e6c9ff38fdcc6867c41037972834dfb4b67884c4"}
Sep 19 08:21:33.280593 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:33.280553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzt4t" event={"ID":"19b17a34-96c9-4497-82d3-e359db7fcf43","Type":"ContainerStarted","Data":"2077a353597a4364e46fe9060d3fd61e542622d43432e352dc39b07cb5461a79"}
Sep 19 08:21:33.280996 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:33.280646    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jzt4t" event={"ID":"19b17a34-96c9-4497-82d3-e359db7fcf43","Type":"ContainerStarted","Data":"196afa5fcd22add9c1bc2cb94e5461bf560759425d357d1f869043b28cb45842"}
Sep 19 08:21:33.280996 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:33.280720    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:33.297831 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:33.297747    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jzt4t" podStartSLOduration=8.687857943000001 podStartE2EDuration="10.297733491s" totalImagesPullingTime="1.609875548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:33.296304437 +0000 UTC m=+45.482496394" watchObservedRunningTime="2026-09-19 08:21:33.297733491 +0000 UTC m=+45.483925444"
Sep 19 08:21:39.140893 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:39.140848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:39.143120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:39.143100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/195ae344-bf5f-4194-9dd0-d36dc7e712ad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4xst4\" (UID: \"195ae344-bf5f-4194-9dd0-d36dc7e712ad\") " pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:39.239288 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:39.239258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4xst4"
Sep 19 08:21:39.504322 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:39.504276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4xst4"]
Sep 19 08:21:40.298706 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:40.298681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xst4" event={"ID":"195ae344-bf5f-4194-9dd0-d36dc7e712ad","Type":"ContainerStarted","Data":"a8c3be6180bd836d5720c6db91be0b715b559977d06cfdd085eb47876b738231"}
Sep 19 08:21:40.298968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:40.298735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xst4" event={"ID":"195ae344-bf5f-4194-9dd0-d36dc7e712ad","Type":"ContainerStarted","Data":"812071afb323b856956c3007c2c10bb70fd4401df9c9136806a1e97ca87db044"}
Sep 19 08:21:40.298968 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:40.298756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xst4" event={"ID":"195ae344-bf5f-4194-9dd0-d36dc7e712ad","Type":"ContainerStarted","Data":"55cc77250ed47d46151576a045a7e430e6c6bbeedd9061c4c56b1062283d6f09"}
Sep 19 08:21:42.309474 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:42.309438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4xst4" event={"ID":"195ae344-bf5f-4194-9dd0-d36dc7e712ad","Type":"ContainerStarted","Data":"d39cecb2a443191f58051a4548349908f2fcf2dc9efb93b9994d8d4cf9a015f7"}
Sep 19 08:21:42.328337 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:42.328288    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4xst4" podStartSLOduration=17.077211203 podStartE2EDuration="19.328273641s" totalImagesPullingTime="2.251062438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:21:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:42.327565973 +0000 UTC m=+54.513757950" watchObservedRunningTime="2026-09-19 08:21:42.328273641 +0000 UTC m=+54.514465598"
Sep 19 08:21:43.285364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.285328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jzt4t"
Sep 19 08:21:43.854916 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.854885    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-5lvxg"]
Sep 19 08:21:43.875355 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.875289    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b48bc997b-sr52z"]
Sep 19 08:21:43.875556 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.875435    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:21:43.879740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.879713    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 19 08:21:43.879900 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.879798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 19 08:21:43.879900 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.879874    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8t52r\""
Sep 19 08:21:43.887366 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.887349    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-5lvxg"]
Sep 19 08:21:43.887472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.887376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b48bc997b-sr52z"]
Sep 19 08:21:43.887472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.887442    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:43.892989 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.892970    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 19 08:21:43.892989 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.892980    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 19 08:21:43.893153 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.893136    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hr2s5\""
Sep 19 08:21:43.893370 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.893350    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 19 08:21:43.901489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.901470    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 19 08:21:43.977464 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:43.977431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7kqv\" (UniqueName: \"kubernetes.io/projected/d37eff43-5018-490b-a02e-74a9156d3e11-kube-api-access-b7kqv\") pod \"downloads-6594f4c75-5lvxg\" (UID: \"d37eff43-5018-490b-a02e-74a9156d3e11\") " pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:21:44.077969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.077936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77669641-85e6-4d4f-a43e-c0082f04e0de-image-registry-private-configuration\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.077984    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7kqv\" (UniqueName: \"kubernetes.io/projected/d37eff43-5018-490b-a02e-74a9156d3e11-kube-api-access-b7kqv\") pod \"downloads-6594f4c75-5lvxg\" (UID: \"d37eff43-5018-490b-a02e-74a9156d3e11\") " pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77669641-85e6-4d4f-a43e-c0082f04e0de-trusted-ca\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77669641-85e6-4d4f-a43e-c0082f04e0de-registry-certificates\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-bound-sa-token\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5zd\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-kube-api-access-sq5zd\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-registry-tls\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77669641-85e6-4d4f-a43e-c0082f04e0de-ca-trust-extracted\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.078281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.078246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77669641-85e6-4d4f-a43e-c0082f04e0de-installation-pull-secrets\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.097001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.096972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7kqv\" (UniqueName: \"kubernetes.io/projected/d37eff43-5018-490b-a02e-74a9156d3e11-kube-api-access-b7kqv\") pod \"downloads-6594f4c75-5lvxg\" (UID: \"d37eff43-5018-490b-a02e-74a9156d3e11\") " pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:21:44.179507 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77669641-85e6-4d4f-a43e-c0082f04e0de-trusted-ca\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179507 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77669641-85e6-4d4f-a43e-c0082f04e0de-registry-certificates\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179507 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-bound-sa-token\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179823 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5zd\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-kube-api-access-sq5zd\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179823 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-registry-tls\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179823 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179752    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77669641-85e6-4d4f-a43e-c0082f04e0de-ca-trust-extracted\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179823 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77669641-85e6-4d4f-a43e-c0082f04e0de-installation-pull-secrets\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.179986 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.179834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77669641-85e6-4d4f-a43e-c0082f04e0de-image-registry-private-configuration\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.180526 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.180492    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/77669641-85e6-4d4f-a43e-c0082f04e0de-registry-certificates\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.180526 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.180513    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/77669641-85e6-4d4f-a43e-c0082f04e0de-ca-trust-extracted\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.180670 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.180545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/77669641-85e6-4d4f-a43e-c0082f04e0de-trusted-ca\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.182267 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.182244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-registry-tls\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.182378 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.182361    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/77669641-85e6-4d4f-a43e-c0082f04e0de-image-registry-private-configuration\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.182463 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.182446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/77669641-85e6-4d4f-a43e-c0082f04e0de-installation-pull-secrets\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.196373 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.196355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:21:44.202396 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.202372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5zd\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-kube-api-access-sq5zd\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.202995 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.202980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/77669641-85e6-4d4f-a43e-c0082f04e0de-bound-sa-token\") pod \"image-registry-b48bc997b-sr52z\" (UID: \"77669641-85e6-4d4f-a43e-c0082f04e0de\") " pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.496101 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.496067    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:44.501417 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.501393    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-5lvxg"]
Sep 19 08:21:44.504159 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:21:44.504139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37eff43_5018_490b_a02e_74a9156d3e11.slice/crio-2c145e4329ca5cbc5452762b0f3e80b21f2e9e308030f005b0d600f3f9d80c4f WatchSource:0}: Error finding container 2c145e4329ca5cbc5452762b0f3e80b21f2e9e308030f005b0d600f3f9d80c4f: Status 404 returned error can't find the container with id 2c145e4329ca5cbc5452762b0f3e80b21f2e9e308030f005b0d600f3f9d80c4f
Sep 19 08:21:44.804345 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:44.804273    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b48bc997b-sr52z"]
Sep 19 08:21:44.807456 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:21:44.807432    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77669641_85e6_4d4f_a43e_c0082f04e0de.slice/crio-faea8a05147dad3573fa2a2f7cb2a0da7bbdcee0c0a179ca583d1592cb404601 WatchSource:0}: Error finding container faea8a05147dad3573fa2a2f7cb2a0da7bbdcee0c0a179ca583d1592cb404601: Status 404 returned error can't find the container with id faea8a05147dad3573fa2a2f7cb2a0da7bbdcee0c0a179ca583d1592cb404601
Sep 19 08:21:45.322341 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:45.322305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b48bc997b-sr52z" event={"ID":"77669641-85e6-4d4f-a43e-c0082f04e0de","Type":"ContainerStarted","Data":"fd1ad6dfbac29ed3ea686bd12dbf995b298a4a5b51a8b736e4d67cf13de837c7"}
Sep 19 08:21:45.322788 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:45.322356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b48bc997b-sr52z" event={"ID":"77669641-85e6-4d4f-a43e-c0082f04e0de","Type":"ContainerStarted","Data":"faea8a05147dad3573fa2a2f7cb2a0da7bbdcee0c0a179ca583d1592cb404601"}
Sep 19 08:21:45.322788 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:45.322388    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:21:45.323595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:45.323574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-5lvxg" event={"ID":"d37eff43-5018-490b-a02e-74a9156d3e11","Type":"ContainerStarted","Data":"2c145e4329ca5cbc5452762b0f3e80b21f2e9e308030f005b0d600f3f9d80c4f"}
Sep 19 08:21:45.355384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:45.355331    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-b48bc997b-sr52z" podStartSLOduration=2.355309792 podStartE2EDuration="2.355309792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:45.354908141 +0000 UTC m=+57.541100110" watchObservedRunningTime="2026-09-19 08:21:45.355309792 +0000 UTC m=+57.541501750"
Sep 19 08:21:47.239261 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:47.239212    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-78n4r"
Sep 19 08:21:51.309682 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.309646    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"]
Sep 19 08:21:51.313258 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.313235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:51.315771 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.315745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 19 08:21:51.315880 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.315821    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9kv2f\""
Sep 19 08:21:51.322519 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.322471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"]
Sep 19 08:21:51.323386 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.323361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7189d561-869e-46ad-b4c8-a14a08ed144b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9sxj6\" (UID: \"7189d561-869e-46ad-b4c8-a14a08ed144b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:51.424132 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.424100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7189d561-869e-46ad-b4c8-a14a08ed144b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9sxj6\" (UID: \"7189d561-869e-46ad-b4c8-a14a08ed144b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:51.424325 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:51.424263    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 19 08:21:51.424390 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:51.424342    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7189d561-869e-46ad-b4c8-a14a08ed144b-tls-certificates podName:7189d561-869e-46ad-b4c8-a14a08ed144b nodeName:}" failed. No retries permitted until 2026-09-19 08:21:51.924322853 +0000 UTC m=+64.110514820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7189d561-869e-46ad-b4c8-a14a08ed144b-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-9sxj6" (UID: "7189d561-869e-46ad-b4c8-a14a08ed144b") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 19 08:21:51.929233 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.929198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7189d561-869e-46ad-b4c8-a14a08ed144b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9sxj6\" (UID: \"7189d561-869e-46ad-b4c8-a14a08ed144b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:51.931914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:51.931891    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7189d561-869e-46ad-b4c8-a14a08ed144b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-9sxj6\" (UID: \"7189d561-869e-46ad-b4c8-a14a08ed144b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:52.223546 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:52.223515    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:52.501662 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:52.501577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"]
Sep 19 08:21:52.505149 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:21:52.505117    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7189d561_869e_46ad_b4c8_a14a08ed144b.slice/crio-c95b3583829d4327049694933a8bafc42a10817b33a2937a2ac6c479af07ed8f WatchSource:0}: Error finding container c95b3583829d4327049694933a8bafc42a10817b33a2937a2ac6c479af07ed8f: Status 404 returned error can't find the container with id c95b3583829d4327049694933a8bafc42a10817b33a2937a2ac6c479af07ed8f
Sep 19 08:21:53.352620 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:53.352589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6" event={"ID":"7189d561-869e-46ad-b4c8-a14a08ed144b","Type":"ContainerStarted","Data":"c95b3583829d4327049694933a8bafc42a10817b33a2937a2ac6c479af07ed8f"}
Sep 19 08:21:54.358049 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:54.358010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6" event={"ID":"7189d561-869e-46ad-b4c8-a14a08ed144b","Type":"ContainerStarted","Data":"49980c60a8999c40ef9ea60277240352e555f3f554f7e6d7a0932826255e5d2f"}
Sep 19 08:21:54.358520 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:54.358257    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:54.363479 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:54.363454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6"
Sep 19 08:21:54.373641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:54.373364    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-9sxj6" podStartSLOduration=2.5064446670000002 podStartE2EDuration="3.373346253s" totalImagesPullingTime="866.901586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:54.372663504 +0000 UTC m=+66.558855473" watchObservedRunningTime="2026-09-19 08:21:54.373346253 +0000 UTC m=+66.559538213"
Sep 19 08:21:55.379558 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.379516    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"]
Sep 19 08:21:55.401413 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.401386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"]
Sep 19 08:21:55.401564 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.401520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.407681 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.407658    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 19 08:21:55.407821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.407658    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 19 08:21:55.407821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.407668    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-khln9\""
Sep 19 08:21:55.407821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.407712    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 19 08:21:55.407821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.407728    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 19 08:21:55.407821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.407668    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 19 08:21:55.455205 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.455166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.455382 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.455246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5qz\" (UniqueName: \"kubernetes.io/projected/651eab1b-6453-4950-8ffe-90f211f8beae-kube-api-access-4z5qz\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.455382 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.455302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/651eab1b-6453-4950-8ffe-90f211f8beae-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.455503 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.455448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.556110 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.556076    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.556110 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.556120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.556383 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.556165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5qz\" (UniqueName: \"kubernetes.io/projected/651eab1b-6453-4950-8ffe-90f211f8beae-kube-api-access-4z5qz\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.556383 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.556195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/651eab1b-6453-4950-8ffe-90f211f8beae-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.556651 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:55.556625    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 19 08:21:55.556746 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:55.556709    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls podName:651eab1b-6453-4950-8ffe-90f211f8beae nodeName:}" failed. No retries permitted until 2026-09-19 08:21:56.056687959 +0000 UTC m=+68.242879898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-h2kt4" (UID: "651eab1b-6453-4950-8ffe-90f211f8beae") : secret "prometheus-operator-tls" not found
Sep 19 08:21:55.557013 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.556992    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/651eab1b-6453-4950-8ffe-90f211f8beae-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.559179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.559158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.570233 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.570202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5qz\" (UniqueName: \"kubernetes.io/projected/651eab1b-6453-4950-8ffe-90f211f8beae-kube-api-access-4z5qz\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:55.758204 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.758167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:55.758427 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.758269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:55.760923 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.760903    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 19 08:21:55.761308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.761288    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 19 08:21:55.771821 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.771792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdfdbe55-3611-49af-8161-9ba335bbc66f-original-pull-secret\") pod \"global-pull-secret-syncer-ttzbh\" (UID: \"bdfdbe55-3611-49af-8161-9ba335bbc66f\") " pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:55.772123 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.772103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/182dd4b2-6f32-484f-9fcc-7bd0de43f465-metrics-certs\") pod \"network-metrics-daemon-dkxrb\" (UID: \"182dd4b2-6f32-484f-9fcc-7bd0de43f465\") " pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:55.960103 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.960066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:55.962715 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.962693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 19 08:21:55.973335 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.973312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 19 08:21:55.983934 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.983908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzlw\" (UniqueName: \"kubernetes.io/projected/8bd653aa-de69-413f-874c-32930fef8b85-kube-api-access-wbzlw\") pod \"network-check-target-jnc2h\" (UID: \"8bd653aa-de69-413f-874c-32930fef8b85\") " pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:55.987865 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.987845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9jjfx\""
Sep 19 08:21:55.995911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.995890    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:21:55.996259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:55.996233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vlr4t\""
Sep 19 08:21:56.001211 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.001191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ttzbh"
Sep 19 08:21:56.004750 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.004734    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkxrb"
Sep 19 08:21:56.061040 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.060959    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:56.061160 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:56.061092    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 19 08:21:56.061160 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:21:56.061155    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls podName:651eab1b-6453-4950-8ffe-90f211f8beae nodeName:}" failed. No retries permitted until 2026-09-19 08:21:57.061141509 +0000 UTC m=+69.247333457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls") pod "prometheus-operator-7bd5d56d6-h2kt4" (UID: "651eab1b-6453-4950-8ffe-90f211f8beae") : secret "prometheus-operator-tls" not found
Sep 19 08:21:56.587259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.587208    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68c69d458b-9zbvv"]
Sep 19 08:21:56.589116 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.589093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.592808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.592749    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 19 08:21:56.592808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.592769    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n9dlx\""
Sep 19 08:21:56.592808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.592784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 19 08:21:56.593088 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.593068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 19 08:21:56.593160 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.593096    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 19 08:21:56.593349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.593335    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 19 08:21:56.602069 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.602041    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c69d458b-9zbvv"]
Sep 19 08:21:56.665001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.664968    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-oauth-config\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.665183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.665012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-config\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.665183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.665050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-service-ca\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.665183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.665080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-oauth-serving-cert\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.665183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.665107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrx5\" (UniqueName: \"kubernetes.io/projected/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-kube-api-access-kwrx5\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.665183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.665127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-serving-cert\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.766465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.766362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-oauth-config\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.766642 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.766539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-config\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.766642 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.766589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-service-ca\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.766756 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.766716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-oauth-serving-cert\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.766815 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.766764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrx5\" (UniqueName: \"kubernetes.io/projected/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-kube-api-access-kwrx5\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.766871 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.766829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-serving-cert\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.767469 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.767442    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-service-ca\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.767594 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.767490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-oauth-serving-cert\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.767594 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.767486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-config\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.769179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.769156    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-oauth-config\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.769617 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.769594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-serving-cert\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.776884 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.776861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrx5\" (UniqueName: \"kubernetes.io/projected/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-kube-api-access-kwrx5\") pod \"console-68c69d458b-9zbvv\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") " pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:56.899672 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:56.899590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:21:57.068503 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:57.068459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:57.071042 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:57.071007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/651eab1b-6453-4950-8ffe-90f211f8beae-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-h2kt4\" (UID: \"651eab1b-6453-4950-8ffe-90f211f8beae\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:21:57.212512 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:21:57.212476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"
Sep 19 08:22:01.988302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:01.988266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79b9897d97-4fh8k"]
Sep 19 08:22:01.991849 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:01.991829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.002706 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.002683    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 19 08:22:02.021008 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.020977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b9897d97-4fh8k"]
Sep 19 08:22:02.112000 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.111872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-oauth-serving-cert\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.112000 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.111952    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-oauth-config\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.112000 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.111983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-trusted-ca-bundle\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.112253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.112006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8289\" (UniqueName: \"kubernetes.io/projected/d564dd2e-89b0-4c8c-8927-d98854ea390f-kube-api-access-f8289\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.112253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.112025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-serving-cert\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.112253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.112040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-service-ca\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.112253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.112149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-config\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-oauth-config\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-trusted-ca-bundle\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8289\" (UniqueName: \"kubernetes.io/projected/d564dd2e-89b0-4c8c-8927-d98854ea390f-kube-api-access-f8289\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-serving-cert\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-service-ca\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-config\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.213882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-oauth-serving-cert\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.214797 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.214720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-oauth-serving-cert\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.215530 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.215482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-trusted-ca-bundle\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.216313 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.215732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-service-ca\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.216313 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.216267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-config\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.217339 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.217308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-oauth-config\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.218918 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.218868    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-serving-cert\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.231975 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.231817    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8289\" (UniqueName: \"kubernetes.io/projected/d564dd2e-89b0-4c8c-8927-d98854ea390f-kube-api-access-f8289\") pod \"console-79b9897d97-4fh8k\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") " pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.303184 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.301716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:02.499629 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.499542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ttzbh"]
Sep 19 08:22:02.501967 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:02.501936    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdfdbe55_3611_49af_8161_9ba335bbc66f.slice/crio-26f7d359769559f99450cc9ebde56e66ab05b48baac4238d8802bffa8af28ea2 WatchSource:0}: Error finding container 26f7d359769559f99450cc9ebde56e66ab05b48baac4238d8802bffa8af28ea2: Status 404 returned error can't find the container with id 26f7d359769559f99450cc9ebde56e66ab05b48baac4238d8802bffa8af28ea2
Sep 19 08:22:02.508193 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.508170    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkxrb"]
Sep 19 08:22:02.511114 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:02.511075    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182dd4b2_6f32_484f_9fcc_7bd0de43f465.slice/crio-6a117591bb5351ffb23691040257d5d0a6e0e016798324a1c5240aaeae11b1c5 WatchSource:0}: Error finding container 6a117591bb5351ffb23691040257d5d0a6e0e016798324a1c5240aaeae11b1c5: Status 404 returned error can't find the container with id 6a117591bb5351ffb23691040257d5d0a6e0e016798324a1c5240aaeae11b1c5
Sep 19 08:22:02.519130 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:02.519009    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd653aa_de69_413f_874c_32930fef8b85.slice/crio-fd69b14323b8abd12a134e7b5a80193b337f75812b2fd6d34954abba08a3c90e WatchSource:0}: Error finding container fd69b14323b8abd12a134e7b5a80193b337f75812b2fd6d34954abba08a3c90e: Status 404 returned error can't find the container with id fd69b14323b8abd12a134e7b5a80193b337f75812b2fd6d34954abba08a3c90e
Sep 19 08:22:02.519773 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:02.519738    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b6ee74_e55f_4b6e_a7d5_8f34b32a0a77.slice/crio-4d6b9f5168c72956156687e96095884c5e0964b510e0e4ea735d35325278ecd1 WatchSource:0}: Error finding container 4d6b9f5168c72956156687e96095884c5e0964b510e0e4ea735d35325278ecd1: Status 404 returned error can't find the container with id 4d6b9f5168c72956156687e96095884c5e0964b510e0e4ea735d35325278ecd1
Sep 19 08:22:02.524035 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.524013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jnc2h"]
Sep 19 08:22:02.525628 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.525605    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68c69d458b-9zbvv"]
Sep 19 08:22:02.526692 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.526671    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4"]
Sep 19 08:22:02.527676 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:02.527644    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651eab1b_6453_4950_8ffe_90f211f8beae.slice/crio-48d2fecdde317c911e4390d18d2c2af9d7234a53c99e9fea7ac39e8bf9bda7ff WatchSource:0}: Error finding container 48d2fecdde317c911e4390d18d2c2af9d7234a53c99e9fea7ac39e8bf9bda7ff: Status 404 returned error can't find the container with id 48d2fecdde317c911e4390d18d2c2af9d7234a53c99e9fea7ac39e8bf9bda7ff
Sep 19 08:22:02.604558 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:02.604518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79b9897d97-4fh8k"]
Sep 19 08:22:02.607654 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:02.607621    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd564dd2e_89b0_4c8c_8927_d98854ea390f.slice/crio-d63b8c0b7db476d59b42907eed94a2f576d6cb83445352fc62444d7635a61050 WatchSource:0}: Error finding container d63b8c0b7db476d59b42907eed94a2f576d6cb83445352fc62444d7635a61050: Status 404 returned error can't find the container with id d63b8c0b7db476d59b42907eed94a2f576d6cb83445352fc62444d7635a61050
Sep 19 08:22:03.416356 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.416248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9897d97-4fh8k" event={"ID":"d564dd2e-89b0-4c8c-8927-d98854ea390f","Type":"ContainerStarted","Data":"d63b8c0b7db476d59b42907eed94a2f576d6cb83445352fc62444d7635a61050"}
Sep 19 08:22:03.441623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.441512    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jnc2h" event={"ID":"8bd653aa-de69-413f-874c-32930fef8b85","Type":"ContainerStarted","Data":"fd69b14323b8abd12a134e7b5a80193b337f75812b2fd6d34954abba08a3c90e"}
Sep 19 08:22:03.458023 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.457919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4" event={"ID":"651eab1b-6453-4950-8ffe-90f211f8beae","Type":"ContainerStarted","Data":"48d2fecdde317c911e4390d18d2c2af9d7234a53c99e9fea7ac39e8bf9bda7ff"}
Sep 19 08:22:03.478808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.478724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c69d458b-9zbvv" event={"ID":"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77","Type":"ContainerStarted","Data":"4d6b9f5168c72956156687e96095884c5e0964b510e0e4ea735d35325278ecd1"}
Sep 19 08:22:03.489981 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.489477    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-5lvxg" event={"ID":"d37eff43-5018-490b-a02e-74a9156d3e11","Type":"ContainerStarted","Data":"afc29e7369b2b3bc290aedbc5ed4af2ba583e8fa6ff4db6cd4d05a099aad94f5"}
Sep 19 08:22:03.489981 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.489856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:22:03.493654 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.493480    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkxrb" event={"ID":"182dd4b2-6f32-484f-9fcc-7bd0de43f465","Type":"ContainerStarted","Data":"6a117591bb5351ffb23691040257d5d0a6e0e016798324a1c5240aaeae11b1c5"}
Sep 19 08:22:03.495574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.495503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ttzbh" event={"ID":"bdfdbe55-3611-49af-8161-9ba335bbc66f","Type":"ContainerStarted","Data":"26f7d359769559f99450cc9ebde56e66ab05b48baac4238d8802bffa8af28ea2"}
Sep 19 08:22:03.502512 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.502489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-5lvxg"
Sep 19 08:22:03.514727 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:03.513633    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-5lvxg" podStartSLOduration=2.674189884 podStartE2EDuration="20.513617115s" totalImagesPullingTime="17.839427231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:03.512421673 +0000 UTC m=+75.698613645" watchObservedRunningTime="2026-09-19 08:22:03.513617115 +0000 UTC m=+75.699809072"
Sep 19 08:22:06.335704 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:06.335211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b48bc997b-sr52z"
Sep 19 08:22:14.560271 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.560168    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jnc2h" event={"ID":"8bd653aa-de69-413f-874c-32930fef8b85","Type":"ContainerStarted","Data":"263ae13395583148dc063d5ecdd6706a83950792450152a3a0f6cdb0d9d06601"}
Sep 19 08:22:14.560716 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.560295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:22:14.562053 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.562029    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4" event={"ID":"651eab1b-6453-4950-8ffe-90f211f8beae","Type":"ContainerStarted","Data":"dae0769319a43085e8a1f5f8273453db0b8b5d67899ed092c07425e822f11586"}
Sep 19 08:22:14.563676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.563647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkxrb" event={"ID":"182dd4b2-6f32-484f-9fcc-7bd0de43f465","Type":"ContainerStarted","Data":"435a7b2aa5de8abf8114a807bbb663fcba68db9f157006b98a779dd114267dac"}
Sep 19 08:22:14.565141 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.565117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ttzbh" event={"ID":"bdfdbe55-3611-49af-8161-9ba335bbc66f","Type":"ContainerStarted","Data":"7b97a2e728da3668f37f9083e556933e43391d782eecea3544aecebe02dd195e"}
Sep 19 08:22:14.575677 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.575628    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jnc2h" podStartSLOduration=72.799635226 podStartE2EDuration="1m24.575610747s" totalImagesPullingTime="11.775975521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:14.574566698 +0000 UTC m=+86.760758657" watchObservedRunningTime="2026-09-19 08:22:14.575610747 +0000 UTC m=+86.761802705"
Sep 19 08:22:14.589611 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:14.589555    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ttzbh" podStartSLOduration=72.784978297 podStartE2EDuration="1m24.589535646s" totalImagesPullingTime="11.804557349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:14.589016986 +0000 UTC m=+86.775208944" watchObservedRunningTime="2026-09-19 08:22:14.589535646 +0000 UTC m=+86.775727604"
Sep 19 08:22:15.570679 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.570643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkxrb" event={"ID":"182dd4b2-6f32-484f-9fcc-7bd0de43f465","Type":"ContainerStarted","Data":"ddd3ba717b1a86d5e936141a8c14368be1de8be64f88ad3e75d898fd1d872a2f"}
Sep 19 08:22:15.572216 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.572189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9897d97-4fh8k" event={"ID":"d564dd2e-89b0-4c8c-8927-d98854ea390f","Type":"ContainerStarted","Data":"82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb"}
Sep 19 08:22:15.573987 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.573962    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4" event={"ID":"651eab1b-6453-4950-8ffe-90f211f8beae","Type":"ContainerStarted","Data":"6efcea5e852bb684334705adbd410483d399f85166e910f26847682dedcc9f6c"}
Sep 19 08:22:15.575563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.575538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c69d458b-9zbvv" event={"ID":"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77","Type":"ContainerStarted","Data":"ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2"}
Sep 19 08:22:15.588982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.588942    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dkxrb" podStartSLOduration=75.196327145 podStartE2EDuration="1m25.588929883s" totalImagesPullingTime="10.392602738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:15.587846336 +0000 UTC m=+87.774038296" watchObservedRunningTime="2026-09-19 08:22:15.588929883 +0000 UTC m=+87.775121839"
Sep 19 08:22:15.605559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.605374    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68c69d458b-9zbvv" podStartSLOduration=7.406009599 podStartE2EDuration="19.60536068s" totalImagesPullingTime="12.199351081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:15.605284525 +0000 UTC m=+87.791476484" watchObservedRunningTime="2026-09-19 08:22:15.60536068 +0000 UTC m=+87.791552638"
Sep 19 08:22:15.622961 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.622905    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-h2kt4" podStartSLOduration=8.858821802 podStartE2EDuration="20.622888746s" totalImagesPullingTime="11.764066944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:15.622189887 +0000 UTC m=+87.808381844" watchObservedRunningTime="2026-09-19 08:22:15.622888746 +0000 UTC m=+87.809080726"
Sep 19 08:22:15.641017 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:15.640957    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79b9897d97-4fh8k" podStartSLOduration=2.515260948 podStartE2EDuration="14.640942154s" totalImagesPullingTime="12.125681206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:15.640069612 +0000 UTC m=+87.826261591" watchObservedRunningTime="2026-09-19 08:22:15.640942154 +0000 UTC m=+87.827134111"
Sep 19 08:22:16.899992 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:16.899958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:22:16.900528 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:16.900006    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:22:16.904670 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:16.904648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:22:17.587592 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.587560    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:22:17.787428 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.787394    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7km6n"]
Sep 19 08:22:17.815416 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.815387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.817702 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.817680    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 19 08:22:17.817702 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.817697    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 19 08:22:17.817922 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.817717    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jtrxn\""
Sep 19 08:22:17.817974 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.817958    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 19 08:22:17.959495 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-sys\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-root\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-textfile\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-wtmp\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-accelerators-collector-config\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdxph\" (UniqueName: \"kubernetes.io/projected/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-kube-api-access-cdxph\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-metrics-client-ca\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:17.959858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:17.959795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-tls\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.060777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-sys\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.060777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-root\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.060956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-textfile\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.060956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060860    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-sys\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.060956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-wtmp\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.060956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-accelerators-collector-config\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060864    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-root\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.060986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-wtmp\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.061040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdxph\" (UniqueName: \"kubernetes.io/projected/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-kube-api-access-cdxph\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.061068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-metrics-client-ca\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.061101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-tls\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061428 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.061070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-textfile\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061535 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.061516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-metrics-client-ca\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.061713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.061688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-accelerators-collector-config\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.063344 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.063324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-tls\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.063432 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.063348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.069947 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.069926    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdxph\" (UniqueName: \"kubernetes.io/projected/eb05e8b9-c157-49e2-b5df-ae7ab0bb6713-kube-api-access-cdxph\") pod \"node-exporter-7km6n\" (UID: \"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713\") " pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.124132 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.124097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7km6n"
Sep 19 08:22:18.163110 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:18.163085    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb05e8b9_c157_49e2_b5df_ae7ab0bb6713.slice/crio-b16f3b5bd28737c33010809e5295ba920c0b3baff0136e9c5d972214b5b2fec6 WatchSource:0}: Error finding container b16f3b5bd28737c33010809e5295ba920c0b3baff0136e9c5d972214b5b2fec6: Status 404 returned error can't find the container with id b16f3b5bd28737c33010809e5295ba920c0b3baff0136e9c5d972214b5b2fec6
Sep 19 08:22:18.592209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:18.592171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7km6n" event={"ID":"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713","Type":"ContainerStarted","Data":"b16f3b5bd28737c33010809e5295ba920c0b3baff0136e9c5d972214b5b2fec6"}
Sep 19 08:22:20.600769 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.600734    2580 generic.go:356] "Generic (PLEG): container finished" podID="eb05e8b9-c157-49e2-b5df-ae7ab0bb6713" containerID="2eabd2fb1aa86beaaa3c8139931dc1713126e8c6c6cfc04cd342ca0cd4e1154a" exitCode=0
Sep 19 08:22:20.601314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.600819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7km6n" event={"ID":"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713","Type":"ContainerDied","Data":"2eabd2fb1aa86beaaa3c8139931dc1713126e8c6c6cfc04cd342ca0cd4e1154a"}
Sep 19 08:22:20.756666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.756643    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"]
Sep 19 08:22:20.783658 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.783634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"]
Sep 19 08:22:20.783802 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.783788    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.786417 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.786394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 19 08:22:20.786535 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.786452    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 19 08:22:20.786535 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.786404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-b28hqfnjvpb73\""
Sep 19 08:22:20.786535 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.786491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 19 08:22:20.786535 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.786527    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 19 08:22:20.786875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.786855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 19 08:22:20.787328 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.787312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-scxp5\""
Sep 19 08:22:20.882631 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-tls\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.882631 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882595    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.882852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.882852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.882852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.883001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47hd\" (UniqueName: \"kubernetes.io/projected/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-kube-api-access-j47hd\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.883001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882897    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-metrics-client-ca\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.883001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.882932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-grpc-tls\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.983866 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.983835    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.983866 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.983868    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984057 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.983960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47hd\" (UniqueName: \"kubernetes.io/projected/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-kube-api-access-j47hd\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984057 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.983996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-metrics-client-ca\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984057 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.984018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-grpc-tls\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984057 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.984045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-tls\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984260 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.984064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984260 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.984111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.984952 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.984901    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-metrics-client-ca\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.986582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.986517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.986686 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.986657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-grpc-tls\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.986860 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.986839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.987066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.986967    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.987066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.987000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.987066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.987054    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-secret-thanos-querier-tls\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:20.993086 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:20.993067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47hd\" (UniqueName: \"kubernetes.io/projected/261fe2e9-ba35-49c2-8b64-4d15ad40f71f-kube-api-access-j47hd\") pod \"thanos-querier-848cfbcf76-qdwb4\" (UID: \"261fe2e9-ba35-49c2-8b64-4d15ad40f71f\") " pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:21.096340 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:21.096307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:21.392639 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:21.392607    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"]
Sep 19 08:22:21.396038 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:21.396009    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261fe2e9_ba35_49c2_8b64_4d15ad40f71f.slice/crio-bcbf4ddbe3ca91ef812f5fc091e5fd2fad5c0b8abc54855d57322f5d00aeaa6f WatchSource:0}: Error finding container bcbf4ddbe3ca91ef812f5fc091e5fd2fad5c0b8abc54855d57322f5d00aeaa6f: Status 404 returned error can't find the container with id bcbf4ddbe3ca91ef812f5fc091e5fd2fad5c0b8abc54855d57322f5d00aeaa6f
Sep 19 08:22:21.609642 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:21.609603    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"bcbf4ddbe3ca91ef812f5fc091e5fd2fad5c0b8abc54855d57322f5d00aeaa6f"}
Sep 19 08:22:21.611842 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:21.611816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7km6n" event={"ID":"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713","Type":"ContainerStarted","Data":"aea7d5aa3cb4fe8271918e3e27c8f951a5a71f9537ef7e8133c1711e91ae4821"}
Sep 19 08:22:21.611966 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:21.611858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7km6n" event={"ID":"eb05e8b9-c157-49e2-b5df-ae7ab0bb6713","Type":"ContainerStarted","Data":"8cff8ce8ccf74d75b01c8f954b849d69f3f885e1826bda234b34c0ff426bebc3"}
Sep 19 08:22:21.631079 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:21.631008    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7km6n" podStartSLOduration=3.031008856 podStartE2EDuration="4.630989916s" totalImagesPullingTime="1.523803358s" totalInitContainerRuntime="76.177702ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:21.630436845 +0000 UTC m=+93.816628799" watchObservedRunningTime="2026-09-19 08:22:21.630989916 +0000 UTC m=+93.817181874"
Sep 19 08:22:22.302909 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.302866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:22.303171 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.302932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:22.308202 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.308173    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:22.526774 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.526743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"]
Sep 19 08:22:22.551319 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.551285    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"]
Sep 19 08:22:22.551319 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.551324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79b9897d97-4fh8k"]
Sep 19 08:22:22.551548 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.551393    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:22.553569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.553513    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-sqskv\""
Sep 19 08:22:22.553702 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.553689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 19 08:22:22.585715 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.583982    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-846578c574-cxxkp"]
Sep 19 08:22:22.602905 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.602876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846578c574-cxxkp"]
Sep 19 08:22:22.603061 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.602985    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.624843 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.624814    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:22.702852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.702813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-console-config\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.702852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.702850    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-oauth-config\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.703372 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rgg\" (UniqueName: \"kubernetes.io/projected/3fcbb230-b4c8-4833-978c-d493b5a29cc0-kube-api-access-j2rgg\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:22.703487 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fcbb230-b4c8-4833-978c-d493b5a29cc0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:22.703487 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkxj\" (UniqueName: \"kubernetes.io/projected/49064a30-13cc-42fa-9235-5d03b5fb3160-kube-api-access-xtkxj\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.703608 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703513    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-oauth-serving-cert\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.703608 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703589    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-serving-cert\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.703710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-service-ca\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.703710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.703657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-trusted-ca-bundle\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.804856 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.804757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rgg\" (UniqueName: \"kubernetes.io/projected/3fcbb230-b4c8-4833-978c-d493b5a29cc0-kube-api-access-j2rgg\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:22.804856 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.804802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fcbb230-b4c8-4833-978c-d493b5a29cc0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:22.805043 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:22:22.804969    2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 19 08:22:22.805043 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:22:22.805028    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcbb230-b4c8-4833-978c-d493b5a29cc0-monitoring-plugin-cert podName:3fcbb230-b4c8-4833-978c-d493b5a29cc0 nodeName:}" failed. No retries permitted until 2026-09-19 08:22:23.30500359 +0000 UTC m=+95.491195525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/3fcbb230-b4c8-4833-978c-d493b5a29cc0-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-dqcj4" (UID: "3fcbb230-b4c8-4833-978c-d493b5a29cc0") : secret "monitoring-plugin-cert" not found
Sep 19 08:22:22.805166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkxj\" (UniqueName: \"kubernetes.io/projected/49064a30-13cc-42fa-9235-5d03b5fb3160-kube-api-access-xtkxj\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.805166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-oauth-serving-cert\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.805166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-serving-cert\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.805166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-service-ca\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.805361 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-trusted-ca-bundle\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.805361 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-console-config\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.805361 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-oauth-config\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.806131 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805935    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-oauth-serving-cert\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.806131 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.805988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-service-ca\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.806345 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.806141    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-console-config\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.806345 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.806271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-trusted-ca-bundle\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.807994 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.807972    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-oauth-config\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.808174 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.808150    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-serving-cert\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.815850 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.815827    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkxj\" (UniqueName: \"kubernetes.io/projected/49064a30-13cc-42fa-9235-5d03b5fb3160-kube-api-access-xtkxj\") pod \"console-846578c574-cxxkp\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") " pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:22.815961 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.815832    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rgg\" (UniqueName: \"kubernetes.io/projected/3fcbb230-b4c8-4833-978c-d493b5a29cc0-kube-api-access-j2rgg\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:22.914791 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:22.914753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:23.309566 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:23.309537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fcbb230-b4c8-4833-978c-d493b5a29cc0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:23.311774 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:23.311747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3fcbb230-b4c8-4833-978c-d493b5a29cc0-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-dqcj4\" (UID: \"3fcbb230-b4c8-4833-978c-d493b5a29cc0\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:23.462138 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:23.462106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:23.558406 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:23.558328    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-846578c574-cxxkp"]
Sep 19 08:22:23.654140 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:23.654103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49064a30_13cc_42fa_9235_5d03b5fb3160.slice/crio-b0f04e0d848f88343e7a652d9170c4b60058094e439287a0ab628a9d87355f79 WatchSource:0}: Error finding container b0f04e0d848f88343e7a652d9170c4b60058094e439287a0ab628a9d87355f79: Status 404 returned error can't find the container with id b0f04e0d848f88343e7a652d9170c4b60058094e439287a0ab628a9d87355f79
Sep 19 08:22:23.953354 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:23.953325    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"]
Sep 19 08:22:23.954852 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:22:23.954822    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcbb230_b4c8_4833_978c_d493b5a29cc0.slice/crio-8cd6a47be7cb43a966218d0215f4c9ee0cd1dd83b479ed334fce1b6053062b17 WatchSource:0}: Error finding container 8cd6a47be7cb43a966218d0215f4c9ee0cd1dd83b479ed334fce1b6053062b17: Status 404 returned error can't find the container with id 8cd6a47be7cb43a966218d0215f4c9ee0cd1dd83b479ed334fce1b6053062b17
Sep 19 08:22:24.632068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.632027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"e67367c65063a8f6f28c8957c5cf1ce4418d50d3b84c72958f4392f8684c7513"}
Sep 19 08:22:24.632481 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.632126    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"605d84719dbba1ec327ed0a11dd345d06ee02cae14bc5a2fa8768d9e6dcbc655"}
Sep 19 08:22:24.632481 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.632209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"129a4e5e1a6be65dc8145d9bc1d934db0e463a345c8684c097cdd8305d2d613b"}
Sep 19 08:22:24.633851 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.633819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4" event={"ID":"3fcbb230-b4c8-4833-978c-d493b5a29cc0","Type":"ContainerStarted","Data":"8cd6a47be7cb43a966218d0215f4c9ee0cd1dd83b479ed334fce1b6053062b17"}
Sep 19 08:22:24.635958 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.635932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846578c574-cxxkp" event={"ID":"49064a30-13cc-42fa-9235-5d03b5fb3160","Type":"ContainerStarted","Data":"855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081"}
Sep 19 08:22:24.636070 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.635977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846578c574-cxxkp" event={"ID":"49064a30-13cc-42fa-9235-5d03b5fb3160","Type":"ContainerStarted","Data":"b0f04e0d848f88343e7a652d9170c4b60058094e439287a0ab628a9d87355f79"}
Sep 19 08:22:24.659827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:24.659765    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-846578c574-cxxkp" podStartSLOduration=2.659744749 podStartE2EDuration="2.659744749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:24.658857675 +0000 UTC m=+96.845049638" watchObservedRunningTime="2026-09-19 08:22:24.659744749 +0000 UTC m=+96.845936708"
Sep 19 08:22:26.645012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.644976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4" event={"ID":"3fcbb230-b4c8-4833-978c-d493b5a29cc0","Type":"ContainerStarted","Data":"b62002ec91392701fd9477c7a3177bb3c77e574c2ea93b13789d5dcfe52497c0"}
Sep 19 08:22:26.645554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.645134    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:26.648480 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.648454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"eb178e60276755ff69a35ee42edd2d7f2276f87b51816afc401c37a2c8271ce2"}
Sep 19 08:22:26.648694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.648521    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"875bf9223aac474b4f196d2ab166e7f3af25c3fbe9bb334ee5e2b81e337df0f1"}
Sep 19 08:22:26.648694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.648561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" event={"ID":"261fe2e9-ba35-49c2-8b64-4d15ad40f71f","Type":"ContainerStarted","Data":"3043c8da77c29b226d8ecb4ab2a51d4e6a6739ade08ba270bd0b378e756a68c5"}
Sep 19 08:22:26.648694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.648603    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:26.650946 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.650929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4"
Sep 19 08:22:26.661569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.661533    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-dqcj4" podStartSLOduration=2.810673946 podStartE2EDuration="4.66151827s" totalImagesPullingTime="1.850844324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:26.659965323 +0000 UTC m=+98.846157279" watchObservedRunningTime="2026-09-19 08:22:26.66151827 +0000 UTC m=+98.847710226"
Sep 19 08:22:26.683272 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:26.682771    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4" podStartSLOduration=2.667984574 podStartE2EDuration="6.682749367s" totalImagesPullingTime="4.014764793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:22:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:26.680518683 +0000 UTC m=+98.866710625" watchObservedRunningTime="2026-09-19 08:22:26.682749367 +0000 UTC m=+98.868941325"
Sep 19 08:22:32.658328 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:32.658297    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-848cfbcf76-qdwb4"
Sep 19 08:22:32.916177 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:32.916085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:32.916848 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:32.916823    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:32.920699 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:32.920678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:33.677911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:33.677877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:22:33.723710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:33.723682    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c69d458b-9zbvv"]
Sep 19 08:22:45.578443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:45.578413    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jnc2h"
Sep 19 08:22:48.641020 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:48.640954    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79b9897d97-4fh8k" podUID="d564dd2e-89b0-4c8c-8927-d98854ea390f" containerName="console" containerID="cri-o://82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb" gracePeriod=15
Sep 19 08:22:48.920173 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:48.920116    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:49.013111 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-service-ca\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-serving-cert\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-oauth-serving-cert\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-config\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013242    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-oauth-config\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8289\" (UniqueName: \"kubernetes.io/projected/d564dd2e-89b0-4c8c-8927-d98854ea390f-kube-api-access-f8289\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-trusted-ca-bundle\") pod \"d564dd2e-89b0-4c8c-8927-d98854ea390f\" (UID: \"d564dd2e-89b0-4c8c-8927-d98854ea390f\") "
Sep 19 08:22:49.013621 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-service-ca" (OuterVolumeSpecName: "service-ca") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:49.013670 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:49.013775 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-config" (OuterVolumeSpecName: "console-config") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:49.013827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.013811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:49.015570 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.015547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:49.015894 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.015865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:49.015894 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.015870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d564dd2e-89b0-4c8c-8927-d98854ea390f-kube-api-access-f8289" (OuterVolumeSpecName: "kube-api-access-f8289") pod "d564dd2e-89b0-4c8c-8927-d98854ea390f" (UID: "d564dd2e-89b0-4c8c-8927-d98854ea390f"). InnerVolumeSpecName "kube-api-access-f8289". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:49.114627 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114593    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-service-ca\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.114627 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114622    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.114627 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114633    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-oauth-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.114834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114642    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.114834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114649    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d564dd2e-89b0-4c8c-8927-d98854ea390f-console-oauth-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.114834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114657    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8289\" (UniqueName: \"kubernetes.io/projected/d564dd2e-89b0-4c8c-8927-d98854ea390f-kube-api-access-f8289\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.114834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.114666    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d564dd2e-89b0-4c8c-8927-d98854ea390f-trusted-ca-bundle\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:49.734959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.734166    2580 generic.go:356] "Generic (PLEG): container finished" podID="d564dd2e-89b0-4c8c-8927-d98854ea390f" containerID="82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb" exitCode=0
Sep 19 08:22:49.734959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.734270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9897d97-4fh8k" event={"ID":"d564dd2e-89b0-4c8c-8927-d98854ea390f","Type":"ContainerDied","Data":"82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb"}
Sep 19 08:22:49.734959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.734347    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79b9897d97-4fh8k" event={"ID":"d564dd2e-89b0-4c8c-8927-d98854ea390f","Type":"ContainerDied","Data":"d63b8c0b7db476d59b42907eed94a2f576d6cb83445352fc62444d7635a61050"}
Sep 19 08:22:49.734959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.734382    2580 scope.go:117] "RemoveContainer" containerID="82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb"
Sep 19 08:22:49.734959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.734576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79b9897d97-4fh8k"
Sep 19 08:22:49.750065 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.750045    2580 scope.go:117] "RemoveContainer" containerID="82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb"
Sep 19 08:22:49.750407 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:22:49.750383    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb\": container with ID starting with 82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb not found: ID does not exist" containerID="82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb"
Sep 19 08:22:49.750480 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.750415    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb"} err="failed to get container status \"82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb\": rpc error: code = NotFound desc = could not find container \"82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb\": container with ID starting with 82755bb619c58e620ea8e025c02f585c8a4badfcb29411c712a43827f5ce69bb not found: ID does not exist"
Sep 19 08:22:49.756066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.756044    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79b9897d97-4fh8k"]
Sep 19 08:22:49.759843 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:49.759820    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79b9897d97-4fh8k"]
Sep 19 08:22:50.079348 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:50.079270    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d564dd2e-89b0-4c8c-8927-d98854ea390f" path="/var/lib/kubelet/pods/d564dd2e-89b0-4c8c-8927-d98854ea390f/volumes"
Sep 19 08:22:58.742729 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:58.742639    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68c69d458b-9zbvv" podUID="c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" containerName="console" containerID="cri-o://ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2" gracePeriod=15
Sep 19 08:22:58.981327 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:58.981304    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:22:59.089113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-serving-cert\") pod \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") "
Sep 19 08:22:59.089113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089065    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-oauth-serving-cert\") pod \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") "
Sep 19 08:22:59.089113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-oauth-config\") pod \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") "
Sep 19 08:22:59.089402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089137    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwrx5\" (UniqueName: \"kubernetes.io/projected/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-kube-api-access-kwrx5\") pod \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") "
Sep 19 08:22:59.089402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089184    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-service-ca\") pod \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") "
Sep 19 08:22:59.089402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-config\") pod \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\" (UID: \"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77\") "
Sep 19 08:22:59.089674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089628    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" (UID: "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:59.089674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-service-ca" (OuterVolumeSpecName: "service-ca") pod "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" (UID: "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:59.089674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.089663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-config" (OuterVolumeSpecName: "console-config") pod "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" (UID: "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:22:59.091265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.091215    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" (UID: "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:59.091421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.091398    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" (UID: "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:22:59.091467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.091446    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-kube-api-access-kwrx5" (OuterVolumeSpecName: "kube-api-access-kwrx5") pod "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" (UID: "c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77"). InnerVolumeSpecName "kube-api-access-kwrx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:22:59.189849 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.189814    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:59.189849 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.189848    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-oauth-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:59.190078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.189862    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-oauth-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:59.190078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.189877    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwrx5\" (UniqueName: \"kubernetes.io/projected/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-kube-api-access-kwrx5\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:59.190078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.189893    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-service-ca\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:59.190078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.189906    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77-console-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:22:59.769666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.769634    2580 generic.go:356] "Generic (PLEG): container finished" podID="c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" containerID="ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2" exitCode=0
Sep 19 08:22:59.770144 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.769687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68c69d458b-9zbvv"
Sep 19 08:22:59.770144 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.769707    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c69d458b-9zbvv" event={"ID":"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77","Type":"ContainerDied","Data":"ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2"}
Sep 19 08:22:59.770144 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.769779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68c69d458b-9zbvv" event={"ID":"c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77","Type":"ContainerDied","Data":"4d6b9f5168c72956156687e96095884c5e0964b510e0e4ea735d35325278ecd1"}
Sep 19 08:22:59.770144 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.769810    2580 scope.go:117] "RemoveContainer" containerID="ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2"
Sep 19 08:22:59.780936 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.780912    2580 scope.go:117] "RemoveContainer" containerID="ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2"
Sep 19 08:22:59.782100 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:22:59.781308    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2\": container with ID starting with ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2 not found: ID does not exist" containerID="ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2"
Sep 19 08:22:59.782100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.781351    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2"} err="failed to get container status \"ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2\": rpc error: code = NotFound desc = could not find container \"ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2\": container with ID starting with ccb82ab05e079059b7167f9b06a408804fa1864296ce5b1cb6627b659642acb2 not found: ID does not exist"
Sep 19 08:22:59.794421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.794398    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68c69d458b-9zbvv"]
Sep 19 08:22:59.800360 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:22:59.800337    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68c69d458b-9zbvv"]
Sep 19 08:23:00.079012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:00.078931    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" path="/var/lib/kubelet/pods/c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77/volumes"
Sep 19 08:23:42.043504 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043468    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c786d67c4-w4blg"]
Sep 19 08:23:42.043949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043758    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" containerName="console"
Sep 19 08:23:42.043949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043772    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" containerName="console"
Sep 19 08:23:42.043949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043780    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d564dd2e-89b0-4c8c-8927-d98854ea390f" containerName="console"
Sep 19 08:23:42.043949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043785    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d564dd2e-89b0-4c8c-8927-d98854ea390f" containerName="console"
Sep 19 08:23:42.043949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043836    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d564dd2e-89b0-4c8c-8927-d98854ea390f" containerName="console"
Sep 19 08:23:42.043949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.043844    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b6ee74-e55f-4b6e-a7d5-8f34b32a0a77" containerName="console"
Sep 19 08:23:42.045858 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.045838    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.048356 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.048336    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 19 08:23:42.048496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.048474    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 19 08:23:42.048620 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.048609    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 19 08:23:42.048749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.048726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 19 08:23:42.048852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.048726    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6bqrv\""
Sep 19 08:23:42.048919 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.048849    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 19 08:23:42.054486 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.054466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 19 08:23:42.059306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.059281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c786d67c4-w4blg"]
Sep 19 08:23:42.218348 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-telemeter-client-tls\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218348 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2tp\" (UniqueName: \"kubernetes.io/projected/5920b672-278d-47ef-aff8-8fa99f507301-kube-api-access-dv2tp\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-metrics-client-ca\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-secret-telemeter-client\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-serving-certs-ca-bundle\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.218764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.218586    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-federate-client-tls\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319426 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319426 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-federate-client-tls\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319426 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-telemeter-client-tls\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2tp\" (UniqueName: \"kubernetes.io/projected/5920b672-278d-47ef-aff8-8fa99f507301-kube-api-access-dv2tp\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-metrics-client-ca\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319602    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-secret-telemeter-client\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.319713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.319644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-serving-certs-ca-bundle\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.320465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.320399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-metrics-client-ca\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.320465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.320441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-serving-certs-ca-bundle\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.321515 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.321487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5920b672-278d-47ef-aff8-8fa99f507301-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.322592 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.322516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-telemeter-client-tls\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.322592 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.322575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-federate-client-tls\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.323057 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.323032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-secret-telemeter-client\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.323714 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.323688    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5920b672-278d-47ef-aff8-8fa99f507301-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.327075 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.327052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2tp\" (UniqueName: \"kubernetes.io/projected/5920b672-278d-47ef-aff8-8fa99f507301-kube-api-access-dv2tp\") pod \"telemeter-client-c786d67c4-w4blg\" (UID: \"5920b672-278d-47ef-aff8-8fa99f507301\") " pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.355776 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.355739    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg"
Sep 19 08:23:42.644175 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.644103    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c786d67c4-w4blg"]
Sep 19 08:23:42.646411 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:23:42.646378    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5920b672_278d_47ef_aff8_8fa99f507301.slice/crio-2c620e4caf635dc0c7eeb0aa6f8a26513f9dfd2e1f8099bde46222986165e3c6 WatchSource:0}: Error finding container 2c620e4caf635dc0c7eeb0aa6f8a26513f9dfd2e1f8099bde46222986165e3c6: Status 404 returned error can't find the container with id 2c620e4caf635dc0c7eeb0aa6f8a26513f9dfd2e1f8099bde46222986165e3c6
Sep 19 08:23:42.922915 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:42.922834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg" event={"ID":"5920b672-278d-47ef-aff8-8fa99f507301","Type":"ContainerStarted","Data":"2c620e4caf635dc0c7eeb0aa6f8a26513f9dfd2e1f8099bde46222986165e3c6"}
Sep 19 08:23:44.933423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:44.933370    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg" event={"ID":"5920b672-278d-47ef-aff8-8fa99f507301","Type":"ContainerStarted","Data":"8f5f7b246dc089bc9b2abc3005535f9d2683c81876247ecac298ebaac01c5110"}
Sep 19 08:23:45.939068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:45.939034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg" event={"ID":"5920b672-278d-47ef-aff8-8fa99f507301","Type":"ContainerStarted","Data":"aa3a793ea0d924c6e5dbc530858e0e8c7f22c7afd069d33d1828cd4463193483"}
Sep 19 08:23:45.939478 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:45.939094    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg" event={"ID":"5920b672-278d-47ef-aff8-8fa99f507301","Type":"ContainerStarted","Data":"10e8936aca775aebe4981dc9cd9c304cc2d87212b4662c1384fa4c9c1e357985"}
Sep 19 08:23:45.961534 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:45.961476    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c786d67c4-w4blg" podStartSLOduration=1.301217443 podStartE2EDuration="3.961456597s" totalImagesPullingTime="2.660239154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:23:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:45.960042403 +0000 UTC m=+178.146234361" watchObservedRunningTime="2026-09-19 08:23:45.961456597 +0000 UTC m=+178.147648554"
Sep 19 08:23:46.048756 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.048718    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-577ccc5bb8-k6tf7"]
Sep 19 08:23:46.051273 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.051216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.065660 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.065634    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577ccc5bb8-k6tf7"]
Sep 19 08:23:46.247055 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247022    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-service-ca\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.247055 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-oauth-config\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.247288 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf5zj\" (UniqueName: \"kubernetes.io/projected/66bbb4f9-712e-43c1-9c83-c870a8c14cea-kube-api-access-nf5zj\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.247288 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-config\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.247288 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247264    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-oauth-serving-cert\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.247379 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247333    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-serving-cert\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.247379 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.247367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-trusted-ca-bundle\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348234 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-serving-cert\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348234 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-trusted-ca-bundle\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348454 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-service-ca\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348454 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-oauth-config\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348557 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348465    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf5zj\" (UniqueName: \"kubernetes.io/projected/66bbb4f9-712e-43c1-9c83-c870a8c14cea-kube-api-access-nf5zj\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348557 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348539    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-config\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.348667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.348638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-oauth-serving-cert\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.349166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.349139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-service-ca\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.349335 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.349198    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-trusted-ca-bundle\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.349335 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.349313    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-oauth-serving-cert\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.349450 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.349366    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-config\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.350833 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.350799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-oauth-config\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.351301 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.351282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-serving-cert\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.356898 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.356877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf5zj\" (UniqueName: \"kubernetes.io/projected/66bbb4f9-712e-43c1-9c83-c870a8c14cea-kube-api-access-nf5zj\") pod \"console-577ccc5bb8-k6tf7\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") " pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.360699 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.360674    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:46.636983 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.636888    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-577ccc5bb8-k6tf7"]
Sep 19 08:23:46.639803 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:23:46.639770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66bbb4f9_712e_43c1_9c83_c870a8c14cea.slice/crio-146de867a3c2922ece5d774782bd42c706940166187d2b4db4a62f5a25334de2 WatchSource:0}: Error finding container 146de867a3c2922ece5d774782bd42c706940166187d2b4db4a62f5a25334de2: Status 404 returned error can't find the container with id 146de867a3c2922ece5d774782bd42c706940166187d2b4db4a62f5a25334de2
Sep 19 08:23:46.944972 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.944926    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577ccc5bb8-k6tf7" event={"ID":"66bbb4f9-712e-43c1-9c83-c870a8c14cea","Type":"ContainerStarted","Data":"2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec"}
Sep 19 08:23:46.945363 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.944985    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577ccc5bb8-k6tf7" event={"ID":"66bbb4f9-712e-43c1-9c83-c870a8c14cea","Type":"ContainerStarted","Data":"146de867a3c2922ece5d774782bd42c706940166187d2b4db4a62f5a25334de2"}
Sep 19 08:23:46.962317 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:46.962272    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-577ccc5bb8-k6tf7" podStartSLOduration=0.962256065 podStartE2EDuration="962.256065ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:23:46.960191621 +0000 UTC m=+179.146383579" watchObservedRunningTime="2026-09-19 08:23:46.962256065 +0000 UTC m=+179.148448022"
Sep 19 08:23:56.361047 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:56.361014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:56.361537 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:56.361105    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:56.365656 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:56.365635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:56.983165 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:56.983136    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:23:57.028308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:23:57.028272    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846578c574-cxxkp"]
Sep 19 08:24:22.051099 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.051055    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-846578c574-cxxkp" podUID="49064a30-13cc-42fa-9235-5d03b5fb3160" containerName="console" containerID="cri-o://855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081" gracePeriod=15
Sep 19 08:24:22.279137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.279114    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:24:22.423279 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-serving-cert\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423279 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-service-ca\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423279 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423257    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtkxj\" (UniqueName: \"kubernetes.io/projected/49064a30-13cc-42fa-9235-5d03b5fb3160-kube-api-access-xtkxj\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423279 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423278    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-oauth-serving-cert\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423304    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-trusted-ca-bundle\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423343    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-oauth-config\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-console-config\") pod \"49064a30-13cc-42fa-9235-5d03b5fb3160\" (UID: \"49064a30-13cc-42fa-9235-5d03b5fb3160\") "
Sep 19 08:24:22.423829 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:22.423829 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423808    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-service-ca" (OuterVolumeSpecName: "service-ca") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:22.423960 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:22.423960 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.423875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-console-config" (OuterVolumeSpecName: "console-config") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:24:22.425520 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.425488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:24:22.425619 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.425539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:24:22.425619 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.425540    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49064a30-13cc-42fa-9235-5d03b5fb3160-kube-api-access-xtkxj" (OuterVolumeSpecName: "kube-api-access-xtkxj") pod "49064a30-13cc-42fa-9235-5d03b5fb3160" (UID: "49064a30-13cc-42fa-9235-5d03b5fb3160"). InnerVolumeSpecName "kube-api-access-xtkxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:24:22.524720 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524681    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-console-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:22.524720 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524713    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:22.524720 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524726    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-service-ca\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:22.524951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524738    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkxj\" (UniqueName: \"kubernetes.io/projected/49064a30-13cc-42fa-9235-5d03b5fb3160-kube-api-access-xtkxj\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:22.524951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524750    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-oauth-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:22.524951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524761    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49064a30-13cc-42fa-9235-5d03b5fb3160-trusted-ca-bundle\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:22.524951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:22.524772    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49064a30-13cc-42fa-9235-5d03b5fb3160-console-oauth-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:24:23.072552 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.072515    2580 generic.go:356] "Generic (PLEG): container finished" podID="49064a30-13cc-42fa-9235-5d03b5fb3160" containerID="855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081" exitCode=0
Sep 19 08:24:23.073011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.072561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-846578c574-cxxkp"
Sep 19 08:24:23.073011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.072566    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846578c574-cxxkp" event={"ID":"49064a30-13cc-42fa-9235-5d03b5fb3160","Type":"ContainerDied","Data":"855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081"}
Sep 19 08:24:23.073011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.072617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-846578c574-cxxkp" event={"ID":"49064a30-13cc-42fa-9235-5d03b5fb3160","Type":"ContainerDied","Data":"b0f04e0d848f88343e7a652d9170c4b60058094e439287a0ab628a9d87355f79"}
Sep 19 08:24:23.073011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.072643    2580 scope.go:117] "RemoveContainer" containerID="855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081"
Sep 19 08:24:23.084674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.084653    2580 scope.go:117] "RemoveContainer" containerID="855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081"
Sep 19 08:24:23.084968 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:24:23.084947    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081\": container with ID starting with 855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081 not found: ID does not exist" containerID="855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081"
Sep 19 08:24:23.085022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.084976    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081"} err="failed to get container status \"855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081\": rpc error: code = NotFound desc = could not find container \"855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081\": container with ID starting with 855f0eb45eed8c65786e5087df1289fd8de66b2817327494dea9ed0bfe247081 not found: ID does not exist"
Sep 19 08:24:23.095948 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.095926    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-846578c574-cxxkp"]
Sep 19 08:24:23.099267 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:23.099247    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-846578c574-cxxkp"]
Sep 19 08:24:24.078175 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:24.078143    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49064a30-13cc-42fa-9235-5d03b5fb3160" path="/var/lib/kubelet/pods/49064a30-13cc-42fa-9235-5d03b5fb3160/volumes"
Sep 19 08:24:58.700181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.700152    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65ffbbc89c-2r849"]
Sep 19 08:24:58.700636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.700452    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49064a30-13cc-42fa-9235-5d03b5fb3160" containerName="console"
Sep 19 08:24:58.700636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.700464    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="49064a30-13cc-42fa-9235-5d03b5fb3160" containerName="console"
Sep 19 08:24:58.700636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.700506    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="49064a30-13cc-42fa-9235-5d03b5fb3160" containerName="console"
Sep 19 08:24:58.704602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.704586    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.718447 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.718420    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65ffbbc89c-2r849"]
Sep 19 08:24:58.797124 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-trusted-ca-bundle\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.797124 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-service-ca\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.797338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-serving-cert\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.797338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-config\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.797338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-oauth-serving-cert\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.797338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797313    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8v4c\" (UniqueName: \"kubernetes.io/projected/d7b8b30f-b239-4b03-b166-d924f8186a5b-kube-api-access-b8v4c\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.797338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.797336    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-oauth-config\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898530    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-oauth-config\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-trusted-ca-bundle\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-service-ca\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-serving-cert\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898646    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-config\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-oauth-serving-cert\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.898764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.898720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8v4c\" (UniqueName: \"kubernetes.io/projected/d7b8b30f-b239-4b03-b166-d924f8186a5b-kube-api-access-b8v4c\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.899435 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.899412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-config\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.899538 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.899465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-service-ca\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.899538 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.899495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-trusted-ca-bundle\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.899538 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.899521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-oauth-serving-cert\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.901136 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.901109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-serving-cert\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.901263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.901137    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-oauth-config\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:58.908418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:58.908398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8v4c\" (UniqueName: \"kubernetes.io/projected/d7b8b30f-b239-4b03-b166-d924f8186a5b-kube-api-access-b8v4c\") pod \"console-65ffbbc89c-2r849\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") " pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:59.013801 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:59.013730    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:24:59.284782 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:24:59.284711    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65ffbbc89c-2r849"]
Sep 19 08:24:59.287682 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:24:59.287658    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b8b30f_b239_4b03_b166_d924f8186a5b.slice/crio-10765ad3862acbfa073f4edd410fec48fde9867065432eff74efc175bb03359c WatchSource:0}: Error finding container 10765ad3862acbfa073f4edd410fec48fde9867065432eff74efc175bb03359c: Status 404 returned error can't find the container with id 10765ad3862acbfa073f4edd410fec48fde9867065432eff74efc175bb03359c
Sep 19 08:25:00.201155 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:00.201124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ffbbc89c-2r849" event={"ID":"d7b8b30f-b239-4b03-b166-d924f8186a5b","Type":"ContainerStarted","Data":"e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1"}
Sep 19 08:25:00.201542 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:00.201169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ffbbc89c-2r849" event={"ID":"d7b8b30f-b239-4b03-b166-d924f8186a5b","Type":"ContainerStarted","Data":"10765ad3862acbfa073f4edd410fec48fde9867065432eff74efc175bb03359c"}
Sep 19 08:25:00.217886 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:00.217847    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65ffbbc89c-2r849" podStartSLOduration=2.217831422 podStartE2EDuration="2.217831422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:00.217500479 +0000 UTC m=+252.403692479" watchObservedRunningTime="2026-09-19 08:25:00.217831422 +0000 UTC m=+252.404023361"
Sep 19 08:25:09.014749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:09.014655    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:25:09.014749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:09.014709    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:25:09.019521 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:09.019499    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:25:09.238368 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:09.238338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:25:09.314386 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:09.314291    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-577ccc5bb8-k6tf7"]
Sep 19 08:25:34.332672 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.332635    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-577ccc5bb8-k6tf7" podUID="66bbb4f9-712e-43c1-9c83-c870a8c14cea" containerName="console" containerID="cri-o://2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec" gracePeriod=15
Sep 19 08:25:34.571381 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.571358    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:25:34.677242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-trusted-ca-bundle\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-service-ca\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677441 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-serving-cert\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-oauth-serving-cert\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677505    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf5zj\" (UniqueName: \"kubernetes.io/projected/66bbb4f9-712e-43c1-9c83-c870a8c14cea-kube-api-access-nf5zj\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677525    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-config\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-service-ca" (OuterVolumeSpecName: "service-ca") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:34.677582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-oauth-config\") pod \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\" (UID: \"66bbb4f9-712e-43c1-9c83-c870a8c14cea\") "
Sep 19 08:25:34.677836 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677593    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:34.677836 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677764    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-trusted-ca-bundle\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:34.677836 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677781    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-service-ca\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:34.677994 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:34.677994 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.677914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-config" (OuterVolumeSpecName: "console-config") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:25:34.679560 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.679538    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:34.679645 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.679626    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:25:34.679687 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.679625    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66bbb4f9-712e-43c1-9c83-c870a8c14cea-kube-api-access-nf5zj" (OuterVolumeSpecName: "kube-api-access-nf5zj") pod "66bbb4f9-712e-43c1-9c83-c870a8c14cea" (UID: "66bbb4f9-712e-43c1-9c83-c870a8c14cea"). InnerVolumeSpecName "kube-api-access-nf5zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:25:34.778306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.778272    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:34.778306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.778300    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-oauth-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:34.778306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.778309    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf5zj\" (UniqueName: \"kubernetes.io/projected/66bbb4f9-712e-43c1-9c83-c870a8c14cea-kube-api-access-nf5zj\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:34.778508 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.778320    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:34.778508 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:34.778329    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66bbb4f9-712e-43c1-9c83-c870a8c14cea-console-oauth-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:25:35.326808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.326772    2580 generic.go:356] "Generic (PLEG): container finished" podID="66bbb4f9-712e-43c1-9c83-c870a8c14cea" containerID="2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec" exitCode=0
Sep 19 08:25:35.326969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.326816    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-577ccc5bb8-k6tf7"
Sep 19 08:25:35.326969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.326860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577ccc5bb8-k6tf7" event={"ID":"66bbb4f9-712e-43c1-9c83-c870a8c14cea","Type":"ContainerDied","Data":"2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec"}
Sep 19 08:25:35.326969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.326908    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-577ccc5bb8-k6tf7" event={"ID":"66bbb4f9-712e-43c1-9c83-c870a8c14cea","Type":"ContainerDied","Data":"146de867a3c2922ece5d774782bd42c706940166187d2b4db4a62f5a25334de2"}
Sep 19 08:25:35.326969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.326931    2580 scope.go:117] "RemoveContainer" containerID="2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec"
Sep 19 08:25:35.335536 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.335354    2580 scope.go:117] "RemoveContainer" containerID="2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec"
Sep 19 08:25:35.335760 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:25:35.335632    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec\": container with ID starting with 2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec not found: ID does not exist" containerID="2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec"
Sep 19 08:25:35.335760 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.335657    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec"} err="failed to get container status \"2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec\": rpc error: code = NotFound desc = could not find container \"2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec\": container with ID starting with 2b347627ebcafe058784bf206fe0c257ed5e5cb728107874db27d8075400a9ec not found: ID does not exist"
Sep 19 08:25:35.347416 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.347390    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-577ccc5bb8-k6tf7"]
Sep 19 08:25:35.350791 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:35.350771    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-577ccc5bb8-k6tf7"]
Sep 19 08:25:36.078909 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:36.078877    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66bbb4f9-712e-43c1-9c83-c870a8c14cea" path="/var/lib/kubelet/pods/66bbb4f9-712e-43c1-9c83-c870a8c14cea/volumes"
Sep 19 08:25:37.216179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.216127    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"]
Sep 19 08:25:37.216555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.216429    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66bbb4f9-712e-43c1-9c83-c870a8c14cea" containerName="console"
Sep 19 08:25:37.216555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.216441    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66bbb4f9-712e-43c1-9c83-c870a8c14cea" containerName="console"
Sep 19 08:25:37.216555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.216495    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66bbb4f9-712e-43c1-9c83-c870a8c14cea" containerName="console"
Sep 19 08:25:37.220504 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.220488    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.222969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.222949    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 19 08:25:37.223096 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.223013    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 19 08:25:37.224207 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.224190    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 19 08:25:37.224312 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.224236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 19 08:25:37.228124 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.228100    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"]
Sep 19 08:25:37.247214 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.247190    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"]
Sep 19 08:25:37.250344 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.250329    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.252767 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.252747    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 19 08:25:37.252859 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.252781    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 19 08:25:37.252859 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.252782    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 19 08:25:37.252947 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.252780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 19 08:25:37.261407 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.261387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"]
Sep 19 08:25:37.296852 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.296827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgrn\" (UniqueName: \"kubernetes.io/projected/9d3702d9-a729-4b7f-b39a-d5b75181057d-kube-api-access-7jgrn\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.296982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.296863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d3702d9-a729-4b7f-b39a-d5b75181057d-tmp\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.296982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.296879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9d3702d9-a729-4b7f-b39a-d5b75181057d-klusterlet-config\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.397431 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d3702d9-a729-4b7f-b39a-d5b75181057d-tmp\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.397431 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9d3702d9-a729-4b7f-b39a-d5b75181057d-klusterlet-config\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.397627 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.397627 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af68da9-d330-4da2-b4e1-36dd41253673-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.397627 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.397740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397640    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-ca\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.397740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397678    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9cr5\" (UniqueName: \"kubernetes.io/projected/7af68da9-d330-4da2-b4e1-36dd41253673-kube-api-access-n9cr5\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.397740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397711    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgrn\" (UniqueName: \"kubernetes.io/projected/9d3702d9-a729-4b7f-b39a-d5b75181057d-kube-api-access-7jgrn\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.397877 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-hub\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.397877 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.397843    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9d3702d9-a729-4b7f-b39a-d5b75181057d-tmp\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.400232 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.400203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9d3702d9-a729-4b7f-b39a-d5b75181057d-klusterlet-config\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.405588 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.405569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgrn\" (UniqueName: \"kubernetes.io/projected/9d3702d9-a729-4b7f-b39a-d5b75181057d-kube-api-access-7jgrn\") pod \"klusterlet-addon-workmgr-d8467fb85-wxvgf\" (UID: \"9d3702d9-a729-4b7f-b39a-d5b75181057d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.498448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.498370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-hub\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.498448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.498414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.498448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.498440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af68da9-d330-4da2-b4e1-36dd41253673-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.498666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.498462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.498666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.498512    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-ca\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.498666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.498547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9cr5\" (UniqueName: \"kubernetes.io/projected/7af68da9-d330-4da2-b4e1-36dd41253673-kube-api-access-n9cr5\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.499251 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.499202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7af68da9-d330-4da2-b4e1-36dd41253673-additional-service-ca\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.500938 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.500908    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.501068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.500988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-ca\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.501142 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.501122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-hub\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.501187 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.501142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7af68da9-d330-4da2-b4e1-36dd41253673-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.507045 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.507024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9cr5\" (UniqueName: \"kubernetes.io/projected/7af68da9-d330-4da2-b4e1-36dd41253673-kube-api-access-n9cr5\") pod \"cluster-proxy-proxy-agent-84bf54f557-7l6zm\" (UID: \"7af68da9-d330-4da2-b4e1-36dd41253673\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.530382 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.530360    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:37.569689 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.569659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"
Sep 19 08:25:37.804136 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.804017    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"]
Sep 19 08:25:37.806669 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:25:37.806644    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3702d9_a729_4b7f_b39a_d5b75181057d.slice/crio-97bc7608614e277b42fcec6b7debb21e97ea521ae3847f7acb3e3bc63f2121b7 WatchSource:0}: Error finding container 97bc7608614e277b42fcec6b7debb21e97ea521ae3847f7acb3e3bc63f2121b7: Status 404 returned error can't find the container with id 97bc7608614e277b42fcec6b7debb21e97ea521ae3847f7acb3e3bc63f2121b7
Sep 19 08:25:37.850675 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:37.850645    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm"]
Sep 19 08:25:37.853380 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:25:37.853355    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7af68da9_d330_4da2_b4e1_36dd41253673.slice/crio-1d09235ba08b60f92e341911567accbd004f7c34f05c5a9385cfcf6950c68753 WatchSource:0}: Error finding container 1d09235ba08b60f92e341911567accbd004f7c34f05c5a9385cfcf6950c68753: Status 404 returned error can't find the container with id 1d09235ba08b60f92e341911567accbd004f7c34f05c5a9385cfcf6950c68753
Sep 19 08:25:38.339713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:38.339665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf" event={"ID":"9d3702d9-a729-4b7f-b39a-d5b75181057d","Type":"ContainerStarted","Data":"97bc7608614e277b42fcec6b7debb21e97ea521ae3847f7acb3e3bc63f2121b7"}
Sep 19 08:25:38.340941 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:38.340909    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm" event={"ID":"7af68da9-d330-4da2-b4e1-36dd41253673","Type":"ContainerStarted","Data":"1d09235ba08b60f92e341911567accbd004f7c34f05c5a9385cfcf6950c68753"}
Sep 19 08:25:43.366658 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:43.366622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf" event={"ID":"9d3702d9-a729-4b7f-b39a-d5b75181057d","Type":"ContainerStarted","Data":"33960a197eabcc5b151068b8ece8a57f6947cfd8d9156aa179cb2d8a94f164e8"}
Sep 19 08:25:43.367028 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:43.366875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:43.368243 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:43.368208    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf"
Sep 19 08:25:43.369002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:43.368974    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm" event={"ID":"7af68da9-d330-4da2-b4e1-36dd41253673","Type":"ContainerStarted","Data":"87a4f61c47b1e95709f9abc79ec90ba1f1c38edf5b2df33b33bbe95ce82f2815"}
Sep 19 08:25:43.383164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:43.383115    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d8467fb85-wxvgf" podStartSLOduration=0.931484312 podStartE2EDuration="6.38309957s" totalImagesPullingTime="5.451615258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:43.382585316 +0000 UTC m=+295.568777274" watchObservedRunningTime="2026-09-19 08:25:43.38309957 +0000 UTC m=+295.569291526"
Sep 19 08:25:44.374415 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:44.374382    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm" event={"ID":"7af68da9-d330-4da2-b4e1-36dd41253673","Type":"ContainerStarted","Data":"9a96100d1b47ef2fefc05ab4611683c2141b1fdca7dc3cb817992ac99b91dbc4"}
Sep 19 08:25:44.374793 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:44.374427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm" event={"ID":"7af68da9-d330-4da2-b4e1-36dd41253673","Type":"ContainerStarted","Data":"e0c05861b11b429e9ea7e7fe60bcc0df6af1915b7a0bcf7c9533636da73485d3"}
Sep 19 08:25:44.392718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:44.392670    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-84bf54f557-7l6zm" podStartSLOduration=1.986963512 podStartE2EDuration="7.392653802s" totalImagesPullingTime="5.40569029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:25:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:25:44.392104213 +0000 UTC m=+296.578296170" watchObservedRunningTime="2026-09-19 08:25:44.392653802 +0000 UTC m=+296.578845759"
Sep 19 08:25:49.992145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:25:49.992115    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 19 08:27:03.588140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.588110    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8697984585-5vbdz"]
Sep 19 08:27:03.590144 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.590127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.603792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.603763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8697984585-5vbdz"]
Sep 19 08:27:03.674089 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-service-ca\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.674253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-serving-cert\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.674253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-trusted-ca-bundle\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.674253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-oauth-config\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.674253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-oauth-serving-cert\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.674253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngq2d\" (UniqueName: \"kubernetes.io/projected/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-kube-api-access-ngq2d\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.674417 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.674256    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-config\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774548 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-oauth-config\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774677 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-oauth-serving-cert\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774677 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngq2d\" (UniqueName: \"kubernetes.io/projected/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-kube-api-access-ngq2d\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774677 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-config\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774677 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-service-ca\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774879 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-serving-cert\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.774879 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.774714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-trusted-ca-bundle\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.775347 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.775325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-oauth-serving-cert\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.775453 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.775356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-service-ca\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.775453 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.775375    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-config\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.775636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.775616    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-trusted-ca-bundle\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.777283 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.777261    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-oauth-config\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.777368 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.777327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-console-serving-cert\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.784694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.784676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngq2d\" (UniqueName: \"kubernetes.io/projected/2230d5ae-7b76-43c2-8bb0-cf989687c6f7-kube-api-access-ngq2d\") pod \"console-8697984585-5vbdz\" (UID: \"2230d5ae-7b76-43c2-8bb0-cf989687c6f7\") " pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:03.898923 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:03.898845    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:04.170423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:04.170400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8697984585-5vbdz"]
Sep 19 08:27:04.172636 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:27:04.172611    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2230d5ae_7b76_43c2_8bb0_cf989687c6f7.slice/crio-601315c738a07445d2dae75b477503c4c3e7d422a5b6bf92edc5d64315df2b66 WatchSource:0}: Error finding container 601315c738a07445d2dae75b477503c4c3e7d422a5b6bf92edc5d64315df2b66: Status 404 returned error can't find the container with id 601315c738a07445d2dae75b477503c4c3e7d422a5b6bf92edc5d64315df2b66
Sep 19 08:27:04.670159 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:04.670125    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8697984585-5vbdz" event={"ID":"2230d5ae-7b76-43c2-8bb0-cf989687c6f7","Type":"ContainerStarted","Data":"820b374e35a4e66b05012c2366af5fda05a4681a9683521ad6855630f353fad1"}
Sep 19 08:27:04.670556 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:04.670172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8697984585-5vbdz" event={"ID":"2230d5ae-7b76-43c2-8bb0-cf989687c6f7","Type":"ContainerStarted","Data":"601315c738a07445d2dae75b477503c4c3e7d422a5b6bf92edc5d64315df2b66"}
Sep 19 08:27:04.689721 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:04.689676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8697984585-5vbdz" podStartSLOduration=1.6896612549999999 podStartE2EDuration="1.689661255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:04.689096152 +0000 UTC m=+376.875288110" watchObservedRunningTime="2026-09-19 08:27:04.689661255 +0000 UTC m=+376.875853212"
Sep 19 08:27:13.899621 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:13.899584    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:13.900064 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:13.899641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:13.904449 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:13.904430    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:14.715735 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:14.715708    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8697984585-5vbdz"
Sep 19 08:27:14.843971 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:14.843936    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65ffbbc89c-2r849"]
Sep 19 08:27:39.862951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:39.862906    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65ffbbc89c-2r849" podUID="d7b8b30f-b239-4b03-b166-d924f8186a5b" containerName="console" containerID="cri-o://e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1" gracePeriod=15
Sep 19 08:27:40.098368 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.098345    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:27:40.175935 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.175844    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-service-ca\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.175935 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.175906    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-oauth-serving-cert\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.176150 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.175941    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8v4c\" (UniqueName: \"kubernetes.io/projected/d7b8b30f-b239-4b03-b166-d924f8186a5b-kube-api-access-b8v4c\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.176150 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.175972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-trusted-ca-bundle\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.176150 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-serving-cert\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.176150 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-oauth-config\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.176150 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-config\") pod \"d7b8b30f-b239-4b03-b166-d924f8186a5b\" (UID: \"d7b8b30f-b239-4b03-b166-d924f8186a5b\") "
Sep 19 08:27:40.176386 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176351    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:40.176430 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176391    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:40.176596 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:40.176660 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.176620    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-config" (OuterVolumeSpecName: "console-config") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:27:40.178262 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.178233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b8b30f-b239-4b03-b166-d924f8186a5b-kube-api-access-b8v4c" (OuterVolumeSpecName: "kube-api-access-b8v4c") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "kube-api-access-b8v4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:27:40.178358 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.178261    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:40.178430 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.178412    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7b8b30f-b239-4b03-b166-d924f8186a5b" (UID: "d7b8b30f-b239-4b03-b166-d924f8186a5b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:27:40.277041 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277013    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-oauth-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.277041 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277036    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8v4c\" (UniqueName: \"kubernetes.io/projected/d7b8b30f-b239-4b03-b166-d924f8186a5b-kube-api-access-b8v4c\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.277041 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277046    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-trusted-ca-bundle\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.277242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277055    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-serving-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.277242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277064    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-oauth-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.277242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277073    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-console-config\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.277242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.277081    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b8b30f-b239-4b03-b166-d924f8186a5b-service-ca\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:27:40.805943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.805907    2580 generic.go:356] "Generic (PLEG): container finished" podID="d7b8b30f-b239-4b03-b166-d924f8186a5b" containerID="e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1" exitCode=0
Sep 19 08:27:40.806095 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.805956    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65ffbbc89c-2r849"
Sep 19 08:27:40.806095 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.805988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ffbbc89c-2r849" event={"ID":"d7b8b30f-b239-4b03-b166-d924f8186a5b","Type":"ContainerDied","Data":"e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1"}
Sep 19 08:27:40.806095 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.806042    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65ffbbc89c-2r849" event={"ID":"d7b8b30f-b239-4b03-b166-d924f8186a5b","Type":"ContainerDied","Data":"10765ad3862acbfa073f4edd410fec48fde9867065432eff74efc175bb03359c"}
Sep 19 08:27:40.806095 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.806062    2580 scope.go:117] "RemoveContainer" containerID="e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1"
Sep 19 08:27:40.814281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.814264    2580 scope.go:117] "RemoveContainer" containerID="e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1"
Sep 19 08:27:40.814537 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:27:40.814518    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1\": container with ID starting with e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1 not found: ID does not exist" containerID="e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1"
Sep 19 08:27:40.814600 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.814547    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1"} err="failed to get container status \"e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1\": rpc error: code = NotFound desc = could not find container \"e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1\": container with ID starting with e5ff975409ba849990974672b16341322eedb97aca264ed543fd9bc1136e65f1 not found: ID does not exist"
Sep 19 08:27:40.826315 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.826292    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65ffbbc89c-2r849"]
Sep 19 08:27:40.830117 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:40.830096    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65ffbbc89c-2r849"]
Sep 19 08:27:42.078882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:27:42.078848    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b8b30f-b239-4b03-b166-d924f8186a5b" path="/var/lib/kubelet/pods/d7b8b30f-b239-4b03-b166-d924f8186a5b/volumes"
Sep 19 08:28:23.147778 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.147695    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r8l8g"]
Sep 19 08:28:23.148240 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.148006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b8b30f-b239-4b03-b166-d924f8186a5b" containerName="console"
Sep 19 08:28:23.148240 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.148018    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b8b30f-b239-4b03-b166-d924f8186a5b" containerName="console"
Sep 19 08:28:23.148240 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.148066    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b8b30f-b239-4b03-b166-d924f8186a5b" containerName="console"
Sep 19 08:28:23.150866 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.150850    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.153102 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.153083    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vpnpf\""
Sep 19 08:28:23.153194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.153087    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 19 08:28:23.153964 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.153948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 19 08:28:23.158774 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.158752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r8l8g"]
Sep 19 08:28:23.208457 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.208424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e-bound-sa-token\") pod \"cert-manager-55b4d954d5-r8l8g\" (UID: \"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e\") " pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.208599 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.208484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z94j\" (UniqueName: \"kubernetes.io/projected/ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e-kube-api-access-4z94j\") pod \"cert-manager-55b4d954d5-r8l8g\" (UID: \"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e\") " pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.308996 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.308954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z94j\" (UniqueName: \"kubernetes.io/projected/ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e-kube-api-access-4z94j\") pod \"cert-manager-55b4d954d5-r8l8g\" (UID: \"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e\") " pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.309139 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.309027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e-bound-sa-token\") pod \"cert-manager-55b4d954d5-r8l8g\" (UID: \"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e\") " pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.317449 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.317422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e-bound-sa-token\") pod \"cert-manager-55b4d954d5-r8l8g\" (UID: \"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e\") " pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.317651 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.317631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z94j\" (UniqueName: \"kubernetes.io/projected/ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e-kube-api-access-4z94j\") pod \"cert-manager-55b4d954d5-r8l8g\" (UID: \"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e\") " pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.459375 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.459355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-r8l8g"
Sep 19 08:28:23.734836 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.734642    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-r8l8g"]
Sep 19 08:28:23.737309 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:28:23.737280    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5e58f3_d1f8_4b95_a5d0_a01a9f080b8e.slice/crio-01449df172cac4a52fa3101e59fc5c75a175eea595402cacbd27ed5cd2c1fc95 WatchSource:0}: Error finding container 01449df172cac4a52fa3101e59fc5c75a175eea595402cacbd27ed5cd2c1fc95: Status 404 returned error can't find the container with id 01449df172cac4a52fa3101e59fc5c75a175eea595402cacbd27ed5cd2c1fc95
Sep 19 08:28:23.739357 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.739339    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:28:23.965107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:23.965076    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-r8l8g" event={"ID":"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e","Type":"ContainerStarted","Data":"01449df172cac4a52fa3101e59fc5c75a175eea595402cacbd27ed5cd2c1fc95"}
Sep 19 08:28:26.978964 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:26.978931    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-r8l8g" event={"ID":"ca5e58f3-d1f8-4b95-a5d0-a01a9f080b8e","Type":"ContainerStarted","Data":"4d1b4bc31bd32f539063b6aa5fb0841919e075817f3b263655b09795e51fbeb4"}
Sep 19 08:28:26.993524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:26.993474    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-r8l8g" podStartSLOduration=1.093812973 podStartE2EDuration="3.993459547s" totalImagesPullingTime="2.899646574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:26.992566889 +0000 UTC m=+459.178758880" watchObservedRunningTime="2026-09-19 08:28:26.993459547 +0000 UTC m=+459.179651503"
Sep 19 08:28:35.215369 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.215334    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"]
Sep 19 08:28:35.218726 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.218704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.221593 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.221568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 19 08:28:35.222467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.222320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 19 08:28:35.222467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.222349    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 19 08:28:35.222467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.222357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 19 08:28:35.222467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.222410    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 19 08:28:35.222733 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.222521    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-czxbn\""
Sep 19 08:28:35.230438 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.230414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"]
Sep 19 08:28:35.312850 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.312810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70be341-8383-4f67-a66a-7664ba2fdde0-cert\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.313006 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.312864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzns8\" (UniqueName: \"kubernetes.io/projected/a70be341-8383-4f67-a66a-7664ba2fdde0-kube-api-access-kzns8\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.313006 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.312926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a70be341-8383-4f67-a66a-7664ba2fdde0-manager-config\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.313006 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.312991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a70be341-8383-4f67-a66a-7664ba2fdde0-metrics-cert\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.414241 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.414186    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70be341-8383-4f67-a66a-7664ba2fdde0-cert\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.414415 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.414262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzns8\" (UniqueName: \"kubernetes.io/projected/a70be341-8383-4f67-a66a-7664ba2fdde0-kube-api-access-kzns8\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.414415 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.414286    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a70be341-8383-4f67-a66a-7664ba2fdde0-manager-config\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.414415 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.414310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a70be341-8383-4f67-a66a-7664ba2fdde0-metrics-cert\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.414993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.414963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a70be341-8383-4f67-a66a-7664ba2fdde0-manager-config\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.416940 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.416916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a70be341-8383-4f67-a66a-7664ba2fdde0-cert\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.417272 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.417253    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a70be341-8383-4f67-a66a-7664ba2fdde0-metrics-cert\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.424999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.424971    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzns8\" (UniqueName: \"kubernetes.io/projected/a70be341-8383-4f67-a66a-7664ba2fdde0-kube-api-access-kzns8\") pod \"lws-controller-manager-6474f95f46-76mq2\" (UID: \"a70be341-8383-4f67-a66a-7664ba2fdde0\") " pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.529293 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.529182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:35.804736 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:35.804660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"]
Sep 19 08:28:35.807852 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:28:35.807830    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70be341_8383_4f67_a66a_7664ba2fdde0.slice/crio-b4093865bd36df5eebc6bb539ae0c8224516a187b3b3b898d4ff4bdfe9e22042 WatchSource:0}: Error finding container b4093865bd36df5eebc6bb539ae0c8224516a187b3b3b898d4ff4bdfe9e22042: Status 404 returned error can't find the container with id b4093865bd36df5eebc6bb539ae0c8224516a187b3b3b898d4ff4bdfe9e22042
Sep 19 08:28:36.011741 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:36.011707    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2" event={"ID":"a70be341-8383-4f67-a66a-7664ba2fdde0","Type":"ContainerStarted","Data":"b4093865bd36df5eebc6bb539ae0c8224516a187b3b3b898d4ff4bdfe9e22042"}
Sep 19 08:28:39.027034 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:39.027001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2" event={"ID":"a70be341-8383-4f67-a66a-7664ba2fdde0","Type":"ContainerStarted","Data":"65be64a38ed4de18f87584b33598a3b283c45d2294a9f3775e642f049ad32d25"}
Sep 19 08:28:39.027403 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:39.027060    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:28:39.042889 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:39.042846    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2" podStartSLOduration=1.605982585 podStartE2EDuration="4.04283158s" totalImagesPullingTime="2.436848995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:39.042034776 +0000 UTC m=+471.228226732" watchObservedRunningTime="2026-09-19 08:28:39.04283158 +0000 UTC m=+471.229023537"
Sep 19 08:28:50.035970 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:28:50.035942    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6474f95f46-76mq2"
Sep 19 08:29:21.410422 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.410389    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"]
Sep 19 08:29:21.418472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.418446    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.421038 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.421009    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 19 08:29:21.421202 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.421184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 19 08:29:21.421386 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.421363    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"]
Sep 19 08:29:21.421444 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.421422    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 19 08:29:21.421984 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.421969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d4hsg\""
Sep 19 08:29:21.422036 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.422024    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 19 08:29:21.491238 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.491185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a82c9f71-9816-4398-bc7a-071a0ce1e361-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.491420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.491247    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fh8\" (UniqueName: \"kubernetes.io/projected/a82c9f71-9816-4398-bc7a-071a0ce1e361-kube-api-access-x6fh8\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.491420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.491327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a82c9f71-9816-4398-bc7a-071a0ce1e361-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.591895 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.591863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a82c9f71-9816-4398-bc7a-071a0ce1e361-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.592077 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.591943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a82c9f71-9816-4398-bc7a-071a0ce1e361-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.592077 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.591971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fh8\" (UniqueName: \"kubernetes.io/projected/a82c9f71-9816-4398-bc7a-071a0ce1e361-kube-api-access-x6fh8\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.592623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.592599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a82c9f71-9816-4398-bc7a-071a0ce1e361-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.594441 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.594418    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a82c9f71-9816-4398-bc7a-071a0ce1e361-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.599395 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.599376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fh8\" (UniqueName: \"kubernetes.io/projected/a82c9f71-9816-4398-bc7a-071a0ce1e361-kube-api-access-x6fh8\") pod \"kuadrant-console-plugin-fbdd4f445-7b6ph\" (UID: \"a82c9f71-9816-4398-bc7a-071a0ce1e361\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:21.728610 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:21.728559    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"
Sep 19 08:29:22.011474 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:22.011451    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph"]
Sep 19 08:29:22.013035 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:29:22.013006    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda82c9f71_9816_4398_bc7a_071a0ce1e361.slice/crio-63b8b1abf1b7a29b7034eb9518fb9c96a4509d2f150261c33f157f329e78a656 WatchSource:0}: Error finding container 63b8b1abf1b7a29b7034eb9518fb9c96a4509d2f150261c33f157f329e78a656: Status 404 returned error can't find the container with id 63b8b1abf1b7a29b7034eb9518fb9c96a4509d2f150261c33f157f329e78a656
Sep 19 08:29:22.192395 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:22.192362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph" event={"ID":"a82c9f71-9816-4398-bc7a-071a0ce1e361","Type":"ContainerStarted","Data":"63b8b1abf1b7a29b7034eb9518fb9c96a4509d2f150261c33f157f329e78a656"}
Sep 19 08:29:27.215868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:27.215835    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph" event={"ID":"a82c9f71-9816-4398-bc7a-071a0ce1e361","Type":"ContainerStarted","Data":"ed97e86d73111574af252e65b64f7c8148129c8f6d937b9d296554caceaaf612"}
Sep 19 08:29:27.233307 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:29:27.233261    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-7b6ph" podStartSLOduration=1.7957952019999999 podStartE2EDuration="6.23324553s" totalImagesPullingTime="4.437450328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:29:27.231793212 +0000 UTC m=+519.417985168" watchObservedRunningTime="2026-09-19 08:29:27.23324553 +0000 UTC m=+519.419437481"
Sep 19 08:30:06.396281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.396248    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljb42"]
Sep 19 08:30:06.399258 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.399242    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:06.401868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.401843    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9jsfz\""
Sep 19 08:30:06.412365 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.412342    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljb42"]
Sep 19 08:30:06.459270 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.459237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq4hv\" (UniqueName: \"kubernetes.io/projected/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8-kube-api-access-kq4hv\") pod \"authorino-7bcfd8f9cd-ljb42\" (UID: \"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:06.539210 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.539179    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-rcnqm"]
Sep 19 08:30:06.542295 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.542280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:06.550664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.550641    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rcnqm"]
Sep 19 08:30:06.559860 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.559833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq4hv\" (UniqueName: \"kubernetes.io/projected/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8-kube-api-access-kq4hv\") pod \"authorino-7bcfd8f9cd-ljb42\" (UID: \"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:06.567853 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.567824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq4hv\" (UniqueName: \"kubernetes.io/projected/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8-kube-api-access-kq4hv\") pod \"authorino-7bcfd8f9cd-ljb42\" (UID: \"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8\") " pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:06.660814 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.660736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5szr2\" (UniqueName: \"kubernetes.io/projected/545166d4-5ac9-4d76-8e82-5b3becf77138-kube-api-access-5szr2\") pod \"authorino-59c974bd48-rcnqm\" (UID: \"545166d4-5ac9-4d76-8e82-5b3becf77138\") " pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:06.707695 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.707666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:06.761604 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.761198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5szr2\" (UniqueName: \"kubernetes.io/projected/545166d4-5ac9-4d76-8e82-5b3becf77138-kube-api-access-5szr2\") pod \"authorino-59c974bd48-rcnqm\" (UID: \"545166d4-5ac9-4d76-8e82-5b3becf77138\") " pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:06.770158 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.770132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5szr2\" (UniqueName: \"kubernetes.io/projected/545166d4-5ac9-4d76-8e82-5b3becf77138-kube-api-access-5szr2\") pod \"authorino-59c974bd48-rcnqm\" (UID: \"545166d4-5ac9-4d76-8e82-5b3becf77138\") " pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:06.851652 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.851619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:06.976625 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:06.976595    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljb42"]
Sep 19 08:30:06.979847 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:30:06.979823    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c40cd2e_9259_4c93_b8ad_d3c7740b01a8.slice/crio-02f63780b3d1b7b67218151163b0dd65cbf7662a29f3725f20320e67a30bc24b WatchSource:0}: Error finding container 02f63780b3d1b7b67218151163b0dd65cbf7662a29f3725f20320e67a30bc24b: Status 404 returned error can't find the container with id 02f63780b3d1b7b67218151163b0dd65cbf7662a29f3725f20320e67a30bc24b
Sep 19 08:30:07.121615 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:07.121587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rcnqm"]
Sep 19 08:30:07.123806 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:30:07.123783    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod545166d4_5ac9_4d76_8e82_5b3becf77138.slice/crio-70c29c81364ac7322f5b10320878d6ab51df4528c62f3f4417f62b3859440b25 WatchSource:0}: Error finding container 70c29c81364ac7322f5b10320878d6ab51df4528c62f3f4417f62b3859440b25: Status 404 returned error can't find the container with id 70c29c81364ac7322f5b10320878d6ab51df4528c62f3f4417f62b3859440b25
Sep 19 08:30:07.364959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:07.364853    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rcnqm" event={"ID":"545166d4-5ac9-4d76-8e82-5b3becf77138","Type":"ContainerStarted","Data":"70c29c81364ac7322f5b10320878d6ab51df4528c62f3f4417f62b3859440b25"}
Sep 19 08:30:07.366014 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:07.365987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42" event={"ID":"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8","Type":"ContainerStarted","Data":"02f63780b3d1b7b67218151163b0dd65cbf7662a29f3725f20320e67a30bc24b"}
Sep 19 08:30:10.380685 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:10.380650    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42" event={"ID":"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8","Type":"ContainerStarted","Data":"bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2"}
Sep 19 08:30:10.396644 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:10.396513    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42" podStartSLOduration=1.650313594 podStartE2EDuration="4.396498289s" totalImagesPullingTime="2.746184695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:10.395553402 +0000 UTC m=+562.581745358" watchObservedRunningTime="2026-09-19 08:30:10.396498289 +0000 UTC m=+562.582690247"
Sep 19 08:30:15.401671 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.401631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rcnqm" event={"ID":"545166d4-5ac9-4d76-8e82-5b3becf77138","Type":"ContainerStarted","Data":"a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12"}
Sep 19 08:30:15.416566 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.416516    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-rcnqm" podStartSLOduration=1.984729951 podStartE2EDuration="9.416500506s" totalImagesPullingTime="7.431770555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:15.415736075 +0000 UTC m=+567.601928043" watchObservedRunningTime="2026-09-19 08:30:15.416500506 +0000 UTC m=+567.602692462"
Sep 19 08:30:15.441811 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.441781    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljb42"]
Sep 19 08:30:15.442008 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.441983    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42" podUID="0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" containerName="authorino" containerID="cri-o://bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2" gracePeriod=30
Sep 19 08:30:15.687142 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.687119    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:15.839351 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.839316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq4hv\" (UniqueName: \"kubernetes.io/projected/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8-kube-api-access-kq4hv\") pod \"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8\" (UID: \"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8\") "
Sep 19 08:30:15.841470 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.841442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8-kube-api-access-kq4hv" (OuterVolumeSpecName: "kube-api-access-kq4hv") pod "0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" (UID: "0c40cd2e-9259-4c93-b8ad-d3c7740b01a8"). InnerVolumeSpecName "kube-api-access-kq4hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:30:15.940169 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:15.940091    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq4hv\" (UniqueName: \"kubernetes.io/projected/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8-kube-api-access-kq4hv\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:30:16.406239 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.406185    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" containerID="bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2" exitCode=0
Sep 19 08:30:16.406700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.406246    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42"
Sep 19 08:30:16.406700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.406246    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42" event={"ID":"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8","Type":"ContainerDied","Data":"bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2"}
Sep 19 08:30:16.406700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.406351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-ljb42" event={"ID":"0c40cd2e-9259-4c93-b8ad-d3c7740b01a8","Type":"ContainerDied","Data":"02f63780b3d1b7b67218151163b0dd65cbf7662a29f3725f20320e67a30bc24b"}
Sep 19 08:30:16.406700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.406367    2580 scope.go:117] "RemoveContainer" containerID="bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2"
Sep 19 08:30:16.414281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.414262    2580 scope.go:117] "RemoveContainer" containerID="bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2"
Sep 19 08:30:16.414568 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:30:16.414551    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2\": container with ID starting with bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2 not found: ID does not exist" containerID="bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2"
Sep 19 08:30:16.414619 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.414580    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2"} err="failed to get container status \"bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2\": rpc error: code = NotFound desc = could not find container \"bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2\": container with ID starting with bb5f4048364f5fec6fa83d12801339167d3168bfe8ca23fdc80a76062e5abbe2 not found: ID does not exist"
Sep 19 08:30:16.420920 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.420895    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljb42"]
Sep 19 08:30:16.426959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:16.426940    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-ljb42"]
Sep 19 08:30:18.079459 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:18.079425    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" path="/var/lib/kubelet/pods/0c40cd2e-9259-4c93-b8ad-d3c7740b01a8/volumes"
Sep 19 08:30:27.529495 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.529459    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-pcjwk"]
Sep 19 08:30:27.529911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.529751    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" containerName="authorino"
Sep 19 08:30:27.529911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.529763    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" containerName="authorino"
Sep 19 08:30:27.529911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.529821    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c40cd2e-9259-4c93-b8ad-d3c7740b01a8" containerName="authorino"
Sep 19 08:30:27.534157 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.534137    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.536646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.536627    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 19 08:30:27.539639 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.539613    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-pcjwk"]
Sep 19 08:30:27.637348 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.637323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7zj\" (UniqueName: \"kubernetes.io/projected/64a9059d-e7a1-4dad-80d2-392b73ad55b6-kube-api-access-7r7zj\") pod \"authorino-6d9898bd99-pcjwk\" (UID: \"64a9059d-e7a1-4dad-80d2-392b73ad55b6\") " pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.637493 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.637390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64a9059d-e7a1-4dad-80d2-392b73ad55b6-tls-cert\") pod \"authorino-6d9898bd99-pcjwk\" (UID: \"64a9059d-e7a1-4dad-80d2-392b73ad55b6\") " pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.738181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.738146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64a9059d-e7a1-4dad-80d2-392b73ad55b6-tls-cert\") pod \"authorino-6d9898bd99-pcjwk\" (UID: \"64a9059d-e7a1-4dad-80d2-392b73ad55b6\") " pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.738377 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.738205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7zj\" (UniqueName: \"kubernetes.io/projected/64a9059d-e7a1-4dad-80d2-392b73ad55b6-kube-api-access-7r7zj\") pod \"authorino-6d9898bd99-pcjwk\" (UID: \"64a9059d-e7a1-4dad-80d2-392b73ad55b6\") " pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.740653 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.740630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64a9059d-e7a1-4dad-80d2-392b73ad55b6-tls-cert\") pod \"authorino-6d9898bd99-pcjwk\" (UID: \"64a9059d-e7a1-4dad-80d2-392b73ad55b6\") " pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.748739 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.748718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7zj\" (UniqueName: \"kubernetes.io/projected/64a9059d-e7a1-4dad-80d2-392b73ad55b6-kube-api-access-7r7zj\") pod \"authorino-6d9898bd99-pcjwk\" (UID: \"64a9059d-e7a1-4dad-80d2-392b73ad55b6\") " pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:27.843898 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:27.843835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-pcjwk"
Sep 19 08:30:28.109487 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:28.109404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-pcjwk"]
Sep 19 08:30:28.112400 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:30:28.112365    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64a9059d_e7a1_4dad_80d2_392b73ad55b6.slice/crio-07ba5b3a005e6e47bd744598990e6e80be8cbd178a2058047ae897c08a890548 WatchSource:0}: Error finding container 07ba5b3a005e6e47bd744598990e6e80be8cbd178a2058047ae897c08a890548: Status 404 returned error can't find the container with id 07ba5b3a005e6e47bd744598990e6e80be8cbd178a2058047ae897c08a890548
Sep 19 08:30:28.457183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:28.457152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-pcjwk" event={"ID":"64a9059d-e7a1-4dad-80d2-392b73ad55b6","Type":"ContainerStarted","Data":"07ba5b3a005e6e47bd744598990e6e80be8cbd178a2058047ae897c08a890548"}
Sep 19 08:30:29.463414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.463381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-pcjwk" event={"ID":"64a9059d-e7a1-4dad-80d2-392b73ad55b6","Type":"ContainerStarted","Data":"81811fe9ff648d914d30a15700791b9b5bba4a4f3be6e2ed2dd51984747941bc"}
Sep 19 08:30:29.483662 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.483615    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-pcjwk" podStartSLOduration=1.982028822 podStartE2EDuration="2.483598609s" totalImagesPullingTime="501.569787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:29.483175717 +0000 UTC m=+581.669367674" watchObservedRunningTime="2026-09-19 08:30:29.483598609 +0000 UTC m=+581.669790573"
Sep 19 08:30:29.510418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.510388    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rcnqm"]
Sep 19 08:30:29.510612 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.510589    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-rcnqm" podUID="545166d4-5ac9-4d76-8e82-5b3becf77138" containerName="authorino" containerID="cri-o://a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12" gracePeriod=30
Sep 19 08:30:29.757454 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.757429    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:29.855626 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.855586    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5szr2\" (UniqueName: \"kubernetes.io/projected/545166d4-5ac9-4d76-8e82-5b3becf77138-kube-api-access-5szr2\") pod \"545166d4-5ac9-4d76-8e82-5b3becf77138\" (UID: \"545166d4-5ac9-4d76-8e82-5b3becf77138\") "
Sep 19 08:30:29.857675 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.857650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545166d4-5ac9-4d76-8e82-5b3becf77138-kube-api-access-5szr2" (OuterVolumeSpecName: "kube-api-access-5szr2") pod "545166d4-5ac9-4d76-8e82-5b3becf77138" (UID: "545166d4-5ac9-4d76-8e82-5b3becf77138"). InnerVolumeSpecName "kube-api-access-5szr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:30:29.956473 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:29.956439    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5szr2\" (UniqueName: \"kubernetes.io/projected/545166d4-5ac9-4d76-8e82-5b3becf77138-kube-api-access-5szr2\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:30:30.468727 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.468691    2580 generic.go:356] "Generic (PLEG): container finished" podID="545166d4-5ac9-4d76-8e82-5b3becf77138" containerID="a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12" exitCode=0
Sep 19 08:30:30.469143 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.468732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rcnqm"
Sep 19 08:30:30.469143 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.468773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rcnqm" event={"ID":"545166d4-5ac9-4d76-8e82-5b3becf77138","Type":"ContainerDied","Data":"a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12"}
Sep 19 08:30:30.469143 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.468813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rcnqm" event={"ID":"545166d4-5ac9-4d76-8e82-5b3becf77138","Type":"ContainerDied","Data":"70c29c81364ac7322f5b10320878d6ab51df4528c62f3f4417f62b3859440b25"}
Sep 19 08:30:30.469143 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.468829    2580 scope.go:117] "RemoveContainer" containerID="a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12"
Sep 19 08:30:30.476962 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.476942    2580 scope.go:117] "RemoveContainer" containerID="a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12"
Sep 19 08:30:30.477258 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:30:30.477238    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12\": container with ID starting with a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12 not found: ID does not exist" containerID="a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12"
Sep 19 08:30:30.477326 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.477270    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12"} err="failed to get container status \"a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12\": rpc error: code = NotFound desc = could not find container \"a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12\": container with ID starting with a7e872463e63985598e9b2ea576942e67f559003668dc772eb03e08cf4e5af12 not found: ID does not exist"
Sep 19 08:30:30.484278 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.484258    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rcnqm"]
Sep 19 08:30:30.491264 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:30.489934    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rcnqm"]
Sep 19 08:30:32.078873 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:32.078839    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545166d4-5ac9-4d76-8e82-5b3becf77138" path="/var/lib/kubelet/pods/545166d4-5ac9-4d76-8e82-5b3becf77138/volumes"
Sep 19 08:30:58.173339 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.173307    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-kbp7n"]
Sep 19 08:30:58.173730 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.173616    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545166d4-5ac9-4d76-8e82-5b3becf77138" containerName="authorino"
Sep 19 08:30:58.173730 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.173630    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="545166d4-5ac9-4d76-8e82-5b3becf77138" containerName="authorino"
Sep 19 08:30:58.173730 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.173703    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="545166d4-5ac9-4d76-8e82-5b3becf77138" containerName="authorino"
Sep 19 08:30:58.180195 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.178634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.181717 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.181693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 19 08:30:58.181859 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.181729    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fqpnd\""
Sep 19 08:30:58.182144 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.182129    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 19 08:30:58.182261 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.182133    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 19 08:30:58.185919 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.185896    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-kbp7n"]
Sep 19 08:30:58.284699 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.284669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4a9eecf-8e1b-441f-9196-33028600a189-data\") pod \"seaweedfs-64568bcd49-kbp7n\" (UID: \"d4a9eecf-8e1b-441f-9196-33028600a189\") " pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.284842 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.284740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7jl\" (UniqueName: \"kubernetes.io/projected/d4a9eecf-8e1b-441f-9196-33028600a189-kube-api-access-zk7jl\") pod \"seaweedfs-64568bcd49-kbp7n\" (UID: \"d4a9eecf-8e1b-441f-9196-33028600a189\") " pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.385164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.385136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7jl\" (UniqueName: \"kubernetes.io/projected/d4a9eecf-8e1b-441f-9196-33028600a189-kube-api-access-zk7jl\") pod \"seaweedfs-64568bcd49-kbp7n\" (UID: \"d4a9eecf-8e1b-441f-9196-33028600a189\") " pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.385293 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.385190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4a9eecf-8e1b-441f-9196-33028600a189-data\") pod \"seaweedfs-64568bcd49-kbp7n\" (UID: \"d4a9eecf-8e1b-441f-9196-33028600a189\") " pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.385553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.385537    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/d4a9eecf-8e1b-441f-9196-33028600a189-data\") pod \"seaweedfs-64568bcd49-kbp7n\" (UID: \"d4a9eecf-8e1b-441f-9196-33028600a189\") " pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.394269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.394241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7jl\" (UniqueName: \"kubernetes.io/projected/d4a9eecf-8e1b-441f-9196-33028600a189-kube-api-access-zk7jl\") pod \"seaweedfs-64568bcd49-kbp7n\" (UID: \"d4a9eecf-8e1b-441f-9196-33028600a189\") " pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.488958 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.488916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:30:58.756533 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:58.756499    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-kbp7n"]
Sep 19 08:30:58.759164 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:30:58.759139    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a9eecf_8e1b_441f_9196_33028600a189.slice/crio-d1d606a995b348d149696be103bed9629b87181d2003ac572f66cd9b895c6020 WatchSource:0}: Error finding container d1d606a995b348d149696be103bed9629b87181d2003ac572f66cd9b895c6020: Status 404 returned error can't find the container with id d1d606a995b348d149696be103bed9629b87181d2003ac572f66cd9b895c6020
Sep 19 08:30:59.589089 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:30:59.589050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-kbp7n" event={"ID":"d4a9eecf-8e1b-441f-9196-33028600a189","Type":"ContainerStarted","Data":"d1d606a995b348d149696be103bed9629b87181d2003ac572f66cd9b895c6020"}
Sep 19 08:31:02.602938 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:31:02.602897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-kbp7n" event={"ID":"d4a9eecf-8e1b-441f-9196-33028600a189","Type":"ContainerStarted","Data":"5b40c56e1189440fbc5790d29f7ad7770b566ac72b315e1625bb525b64acd82b"}
Sep 19 08:31:02.603336 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:31:02.602965    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:31:02.635713 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:31:02.635665    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-kbp7n" podStartSLOduration=1.668567897 podStartE2EDuration="4.63564802s" totalImagesPullingTime="2.967080123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:31:02.634816778 +0000 UTC m=+614.821008734" watchObservedRunningTime="2026-09-19 08:31:02.63564802 +0000 UTC m=+614.821839976"
Sep 19 08:31:08.615101 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:31:08.615065    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-kbp7n"
Sep 19 08:34:19.762580 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.762500    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"]
Sep 19 08:34:19.765801 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.765784    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:19.769266 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.769244    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 19 08:34:19.769394 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.769260    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-tms8v\""
Sep 19 08:34:19.769394 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.769271    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 19 08:34:19.769394 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.769351    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 19 08:34:19.776899 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.776876    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"]
Sep 19 08:34:19.937562 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.937519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:19.937721 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.937580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:19.937721 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.937633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:19.937721 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.937694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8987a644-d51d-4600-a335-b138324e511f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:19.937819 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.937739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscbr\" (UniqueName: \"kubernetes.io/projected/8987a644-d51d-4600-a335-b138324e511f-kube-api-access-rscbr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:19.937819 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:19.937764    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.038611 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.038611 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038579    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.038824 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038616    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.038824 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8987a644-d51d-4600-a335-b138324e511f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.038824 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscbr\" (UniqueName: \"kubernetes.io/projected/8987a644-d51d-4600-a335-b138324e511f-kube-api-access-rscbr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.038824 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.039036 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.038975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.039095 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.039072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.039208 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.039191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.041670 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.041647    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.041802 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.041764    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8987a644-d51d-4600-a335-b138324e511f-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.051552 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.051518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscbr\" (UniqueName: \"kubernetes.io/projected/8987a644-d51d-4600-a335-b138324e511f-kube-api-access-rscbr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.075559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.075520    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:20.347187 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.347114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"]
Sep 19 08:34:20.350394 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:34:20.350371    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8987a644_d51d_4600_a335_b138324e511f.slice/crio-9cffcd1bec719180c3edf86f199db55a173f7c77ec6fc8604a6e659eb0447046 WatchSource:0}: Error finding container 9cffcd1bec719180c3edf86f199db55a173f7c77ec6fc8604a6e659eb0447046: Status 404 returned error can't find the container with id 9cffcd1bec719180c3edf86f199db55a173f7c77ec6fc8604a6e659eb0447046
Sep 19 08:34:20.352167 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.352151    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:34:20.369006 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:20.368976    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" event={"ID":"8987a644-d51d-4600-a335-b138324e511f","Type":"ContainerStarted","Data":"9cffcd1bec719180c3edf86f199db55a173f7c77ec6fc8604a6e659eb0447046"}
Sep 19 08:34:22.378927 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:22.378890    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" event={"ID":"8987a644-d51d-4600-a335-b138324e511f","Type":"ContainerStarted","Data":"6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"}
Sep 19 08:34:22.397850 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:22.397794    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" podStartSLOduration=1.909451284 podStartE2EDuration="3.397730118s" totalImagesPullingTime="1.488278834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:22.397521995 +0000 UTC m=+814.583713980" watchObservedRunningTime="2026-09-19 08:34:22.397730118 +0000 UTC m=+814.583922075"
Sep 19 08:34:23.075961 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:23.075927    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:30.078473 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:30.078444    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:30.088284 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:30.088264    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:30.419746 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:30.419663    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:59.415090 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.415057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"]
Sep 19 08:34:59.423943 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:34:59.423902    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:34:59.423943 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:34:59.423943 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:34:59.423943 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828" cmd=["/bin/sleep","15"]
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.423968    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" message="" err=<
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:34:59.423984    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" podUID="8987a644-d51d-4600-a335-b138324e511f" containerName="main" containerID="cri-o://6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"
Sep 19 08:34:59.424184 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.424028    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" podUID="8987a644-d51d-4600-a335-b138324e511f" containerName="main" containerID="cri-o://6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828" gracePeriod=60
Sep 19 08:34:59.767409 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.767383    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:34:59.882092 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rscbr\" (UniqueName: \"kubernetes.io/projected/8987a644-d51d-4600-a335-b138324e511f-kube-api-access-rscbr\") pod \"8987a644-d51d-4600-a335-b138324e511f\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") "
Sep 19 08:34:59.882282 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882118    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8987a644-d51d-4600-a335-b138324e511f-tls-certs\") pod \"8987a644-d51d-4600-a335-b138324e511f\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") "
Sep 19 08:34:59.882282 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882160    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-tmp-dir\") pod \"8987a644-d51d-4600-a335-b138324e511f\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") "
Sep 19 08:34:59.882282 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882187    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-home\") pod \"8987a644-d51d-4600-a335-b138324e511f\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") "
Sep 19 08:34:59.882282 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882210    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-dshm\") pod \"8987a644-d51d-4600-a335-b138324e511f\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") "
Sep 19 08:34:59.882516 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882381    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-model-cache\") pod \"8987a644-d51d-4600-a335-b138324e511f\" (UID: \"8987a644-d51d-4600-a335-b138324e511f\") "
Sep 19 08:34:59.882516 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-home" (OuterVolumeSpecName: "home") pod "8987a644-d51d-4600-a335-b138324e511f" (UID: "8987a644-d51d-4600-a335-b138324e511f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:59.882516 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8987a644-d51d-4600-a335-b138324e511f" (UID: "8987a644-d51d-4600-a335-b138324e511f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:59.882650 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8987a644-d51d-4600-a335-b138324e511f" (UID: "8987a644-d51d-4600-a335-b138324e511f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:59.882704 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882682    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:34:59.882704 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.882701    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:34:59.884477 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.884452    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8987a644-d51d-4600-a335-b138324e511f-kube-api-access-rscbr" (OuterVolumeSpecName: "kube-api-access-rscbr") pod "8987a644-d51d-4600-a335-b138324e511f" (UID: "8987a644-d51d-4600-a335-b138324e511f"). InnerVolumeSpecName "kube-api-access-rscbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:34:59.884661 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.884645    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-dshm" (OuterVolumeSpecName: "dshm") pod "8987a644-d51d-4600-a335-b138324e511f" (UID: "8987a644-d51d-4600-a335-b138324e511f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:34:59.884707 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.884683    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8987a644-d51d-4600-a335-b138324e511f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8987a644-d51d-4600-a335-b138324e511f" (UID: "8987a644-d51d-4600-a335-b138324e511f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:34:59.983574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.983538    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:34:59.983574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.983569    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rscbr\" (UniqueName: \"kubernetes.io/projected/8987a644-d51d-4600-a335-b138324e511f-kube-api-access-rscbr\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:34:59.983574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.983580    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8987a644-d51d-4600-a335-b138324e511f-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:34:59.983796 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:34:59.983589    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8987a644-d51d-4600-a335-b138324e511f-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:35:00.538514 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.538479    2580 generic.go:356] "Generic (PLEG): container finished" podID="8987a644-d51d-4600-a335-b138324e511f" containerID="6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828" exitCode=0
Sep 19 08:35:00.538941 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.538534    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"
Sep 19 08:35:00.538941 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.538546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" event={"ID":"8987a644-d51d-4600-a335-b138324e511f","Type":"ContainerDied","Data":"6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"}
Sep 19 08:35:00.538941 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.538613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8" event={"ID":"8987a644-d51d-4600-a335-b138324e511f","Type":"ContainerDied","Data":"9cffcd1bec719180c3edf86f199db55a173f7c77ec6fc8604a6e659eb0447046"}
Sep 19 08:35:00.538941 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.538635    2580 scope.go:117] "RemoveContainer" containerID="6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"
Sep 19 08:35:00.547058 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.547040    2580 scope.go:117] "RemoveContainer" containerID="6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"
Sep 19 08:35:00.547437 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:35:00.547415    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828\": container with ID starting with 6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828 not found: ID does not exist" containerID="6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"
Sep 19 08:35:00.547496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.547444    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828"} err="failed to get container status \"6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828\": rpc error: code = NotFound desc = could not find container \"6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828\": container with ID starting with 6c971db69e118320106fa3216466721412ba0f2c7d92dde2a09e0ad8ff2c4828 not found: ID does not exist"
Sep 19 08:35:00.553924 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.553901    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"]
Sep 19 08:35:00.557315 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:00.557292    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94f65rx8"]
Sep 19 08:35:02.079371 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:02.079341    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8987a644-d51d-4600-a335-b138324e511f" path="/var/lib/kubelet/pods/8987a644-d51d-4600-a335-b138324e511f/volumes"
Sep 19 08:35:54.857818 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.857788    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"]
Sep 19 08:35:54.858265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.858096    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8987a644-d51d-4600-a335-b138324e511f" containerName="main"
Sep 19 08:35:54.858265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.858106    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8987a644-d51d-4600-a335-b138324e511f" containerName="main"
Sep 19 08:35:54.858265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.858161    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8987a644-d51d-4600-a335-b138324e511f" containerName="main"
Sep 19 08:35:54.862392 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.862372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:54.864837 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.864820    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 19 08:35:54.865725 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.865705    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-bqpjp\""
Sep 19 08:35:54.865798 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.865714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 19 08:35:54.865798 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.865754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 19 08:35:54.870488 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.870465    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"]
Sep 19 08:35:54.913475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.913442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9brw\" (UniqueName: \"kubernetes.io/projected/5598bfd7-af5b-4b48-911b-4b1baf89922a-kube-api-access-d9brw\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:54.913655 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.913512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:54.913655 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.913538    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:54.913655 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.913571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5598bfd7-af5b-4b48-911b-4b1baf89922a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:54.913655 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.913608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:54.913837 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:54.913699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9brw\" (UniqueName: \"kubernetes.io/projected/5598bfd7-af5b-4b48-911b-4b1baf89922a-kube-api-access-d9brw\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5598bfd7-af5b-4b48-911b-4b1baf89922a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014607 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.014728 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.014711    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.016501 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.016480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.016870 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.016854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5598bfd7-af5b-4b48-911b-4b1baf89922a-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.022139 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.022117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9brw\" (UniqueName: \"kubernetes.io/projected/5598bfd7-af5b-4b48-911b-4b1baf89922a-kube-api-access-d9brw\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.172164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.172082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:35:55.445763 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.445735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"]
Sep 19 08:35:55.448238 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:35:55.448202    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5598bfd7_af5b_4b48_911b_4b1baf89922a.slice/crio-10f3cdb3812cd172acb31beb81c50b4534b728c6882c55f26e8de8359e9c17cb WatchSource:0}: Error finding container 10f3cdb3812cd172acb31beb81c50b4534b728c6882c55f26e8de8359e9c17cb: Status 404 returned error can't find the container with id 10f3cdb3812cd172acb31beb81c50b4534b728c6882c55f26e8de8359e9c17cb
Sep 19 08:35:55.757333 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.757251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" event={"ID":"5598bfd7-af5b-4b48-911b-4b1baf89922a","Type":"ContainerStarted","Data":"5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6"}
Sep 19 08:35:55.757481 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.757343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" event={"ID":"5598bfd7-af5b-4b48-911b-4b1baf89922a","Type":"ContainerStarted","Data":"10f3cdb3812cd172acb31beb81c50b4534b728c6882c55f26e8de8359e9c17cb"}
Sep 19 08:35:55.776085 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:55.776025    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" podStartSLOduration=1.776001985 podStartE2EDuration="1.776001985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:35:55.775142906 +0000 UTC m=+907.961334863" watchObservedRunningTime="2026-09-19 08:35:55.776001985 +0000 UTC m=+907.962193942"
Sep 19 08:35:56.173258 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:35:56.173161    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:36:05.173468 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:05.173429    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:36:05.185837 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:05.185812    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:36:05.808640 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:05.808608    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:36:36.798402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.798367    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"]
Sep 19 08:36:36.801502 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.801485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:36.804102 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.804079    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Sep 19 08:36:36.805071 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.805053    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cpu-accelerator-preset-ser-afc820a4-epp-sa-dockercfg-m8rp4\""
Sep 19 08:36:36.805364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.805341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-nk589\""
Sep 19 08:36:36.805364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.805356    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Sep 19 08:36:36.805515 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.805371    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Sep 19 08:36:36.810775 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.810749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"]
Sep 19 08:36:36.971945 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.971914    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb26f\" (UniqueName: \"kubernetes.io/projected/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-kube-api-access-qb26f\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:36.972125 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:36.971973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:37.073016 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.072924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb26f\" (UniqueName: \"kubernetes.io/projected/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-kube-api-access-qb26f\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:37.073016 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.072994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:37.075375 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.075356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:37.081247 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.081195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb26f\" (UniqueName: \"kubernetes.io/projected/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-kube-api-access-qb26f\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:37.112070 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.112047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:37.381808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.381783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"]
Sep 19 08:36:37.384390 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:36:37.384361    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf584e61e_bf1a_46b1_afa3_b7d426ee77ad.slice/crio-4367b52f9c5cbf1a8a26fced59b9a9e037b70658bc75fc9493416c147a3ec541 WatchSource:0}: Error finding container 4367b52f9c5cbf1a8a26fced59b9a9e037b70658bc75fc9493416c147a3ec541: Status 404 returned error can't find the container with id 4367b52f9c5cbf1a8a26fced59b9a9e037b70658bc75fc9493416c147a3ec541
Sep 19 08:36:37.927093 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:37.927056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" event={"ID":"f584e61e-bf1a-46b1-afa3-b7d426ee77ad","Type":"ContainerStarted","Data":"4367b52f9c5cbf1a8a26fced59b9a9e037b70658bc75fc9493416c147a3ec541"}
Sep 19 08:36:40.945885 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:40.945850    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" event={"ID":"f584e61e-bf1a-46b1-afa3-b7d426ee77ad","Type":"ContainerStarted","Data":"7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39"}
Sep 19 08:36:40.946312 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:40.945947    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:36:40.963042 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:40.962995    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" podStartSLOduration=2.447366708 podStartE2EDuration="4.962980872s" totalImagesPullingTime="2.515614164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:36:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:36:40.961126509 +0000 UTC m=+953.147318465" watchObservedRunningTime="2026-09-19 08:36:40.962980872 +0000 UTC m=+953.149172829"
Sep 19 08:36:59.708562 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:59.708531    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"]
Sep 19 08:36:59.717471 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:36:59.717438    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:36:59.717471 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:36:59.717471 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:36:59.717471 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6" cmd=["/bin/sleep","15"]
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:59.717486    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" message="" err=<
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:36:59.717499    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerName="main" containerID="cri-o://5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6"
Sep 19 08:36:59.717727 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:59.717545    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerName="main" containerID="cri-o://5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6" gracePeriod=60
Sep 19 08:36:59.798637 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:36:59.798586    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Sep 19 08:37:00.026669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.026640    2580 generic.go:356] "Generic (PLEG): container finished" podID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerID="5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6" exitCode=0
Sep 19 08:37:00.026829 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.026721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" event={"ID":"5598bfd7-af5b-4b48-911b-4b1baf89922a","Type":"ContainerDied","Data":"5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6"}
Sep 19 08:37:00.059767 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.059744    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:37:00.161066 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-dshm\") pod \"5598bfd7-af5b-4b48-911b-4b1baf89922a\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") "
Sep 19 08:37:00.161265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5598bfd7-af5b-4b48-911b-4b1baf89922a-tls-certs\") pod \"5598bfd7-af5b-4b48-911b-4b1baf89922a\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") "
Sep 19 08:37:00.161265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161128    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9brw\" (UniqueName: \"kubernetes.io/projected/5598bfd7-af5b-4b48-911b-4b1baf89922a-kube-api-access-d9brw\") pod \"5598bfd7-af5b-4b48-911b-4b1baf89922a\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") "
Sep 19 08:37:00.161265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-tmp-dir\") pod \"5598bfd7-af5b-4b48-911b-4b1baf89922a\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") "
Sep 19 08:37:00.161437 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161293    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-model-cache\") pod \"5598bfd7-af5b-4b48-911b-4b1baf89922a\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") "
Sep 19 08:37:00.161437 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-home\") pod \"5598bfd7-af5b-4b48-911b-4b1baf89922a\" (UID: \"5598bfd7-af5b-4b48-911b-4b1baf89922a\") "
Sep 19 08:37:00.161532 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5598bfd7-af5b-4b48-911b-4b1baf89922a" (UID: "5598bfd7-af5b-4b48-911b-4b1baf89922a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:00.161589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5598bfd7-af5b-4b48-911b-4b1baf89922a" (UID: "5598bfd7-af5b-4b48-911b-4b1baf89922a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:00.161640 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-home" (OuterVolumeSpecName: "home") pod "5598bfd7-af5b-4b48-911b-4b1baf89922a" (UID: "5598bfd7-af5b-4b48-911b-4b1baf89922a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:00.161640 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161632    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:37:00.161746 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.161650    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:37:00.163373 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.163343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-dshm" (OuterVolumeSpecName: "dshm") pod "5598bfd7-af5b-4b48-911b-4b1baf89922a" (UID: "5598bfd7-af5b-4b48-911b-4b1baf89922a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:37:00.163495 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.163427    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5598bfd7-af5b-4b48-911b-4b1baf89922a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5598bfd7-af5b-4b48-911b-4b1baf89922a" (UID: "5598bfd7-af5b-4b48-911b-4b1baf89922a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:37:00.163495 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.163450    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5598bfd7-af5b-4b48-911b-4b1baf89922a-kube-api-access-d9brw" (OuterVolumeSpecName: "kube-api-access-d9brw") pod "5598bfd7-af5b-4b48-911b-4b1baf89922a" (UID: "5598bfd7-af5b-4b48-911b-4b1baf89922a"). InnerVolumeSpecName "kube-api-access-d9brw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:37:00.262950 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.262880    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:37:00.262950 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.262901    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5598bfd7-af5b-4b48-911b-4b1baf89922a-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:37:00.262950 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.262910    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5598bfd7-af5b-4b48-911b-4b1baf89922a-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:37:00.262950 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:00.262919    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9brw\" (UniqueName: \"kubernetes.io/projected/5598bfd7-af5b-4b48-911b-4b1baf89922a-kube-api-access-d9brw\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:37:01.036443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:01.036417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"
Sep 19 08:37:01.036883 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:01.036416    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h" event={"ID":"5598bfd7-af5b-4b48-911b-4b1baf89922a","Type":"ContainerDied","Data":"10f3cdb3812cd172acb31beb81c50b4534b728c6882c55f26e8de8359e9c17cb"}
Sep 19 08:37:01.036883 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:01.036572    2580 scope.go:117] "RemoveContainer" containerID="5fab1a4f266fbc50697f91ba973bce8c6f08b04918b85bb712e68a43d6f71ac6"
Sep 19 08:37:01.057519 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:01.057492    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"]
Sep 19 08:37:01.061838 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:01.061815    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-x7h6h"]
Sep 19 08:37:02.078939 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:02.078906    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" path="/var/lib/kubelet/pods/5598bfd7-af5b-4b48-911b-4b1baf89922a/volumes"
Sep 19 08:37:11.951415 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:37:11.951387    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:38:03.064831 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.064795    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"]
Sep 19 08:38:03.067724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.065734    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerName="main"
Sep 19 08:38:03.067724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.065772    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerName="main"
Sep 19 08:38:03.067724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.066062    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5598bfd7-af5b-4b48-911b-4b1baf89922a" containerName="main"
Sep 19 08:38:03.073774 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.073748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.077527 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.077504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-mn77q\""
Sep 19 08:38:03.077666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.077593    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 19 08:38:03.077666 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.077631    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-cwklf\""
Sep 19 08:38:03.077790 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.077710    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 19 08:38:03.077790 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.077777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 19 08:38:03.081343 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.081320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"]
Sep 19 08:38:03.174194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.174165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpd2p\" (UniqueName: \"kubernetes.io/projected/5b9ff759-ad0e-4d35-a354-e165df7b9152-kube-api-access-cpd2p\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.174194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.174198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9ff759-ad0e-4d35-a354-e165df7b9152-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.275451 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.275417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpd2p\" (UniqueName: \"kubernetes.io/projected/5b9ff759-ad0e-4d35-a354-e165df7b9152-kube-api-access-cpd2p\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.275591 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.275457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9ff759-ad0e-4d35-a354-e165df7b9152-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.277911 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.277887    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9ff759-ad0e-4d35-a354-e165df7b9152-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.283718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.283692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpd2p\" (UniqueName: \"kubernetes.io/projected/5b9ff759-ad0e-4d35-a354-e165df7b9152-kube-api-access-cpd2p\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.384782 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.384711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:03.657636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:03.657559    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"]
Sep 19 08:38:03.661313 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:38:03.661288    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9ff759_ad0e_4d35_a354_e165df7b9152.slice/crio-01d55d36e0dae09a8d1e2f6eb15f8d8b2ef7bc7c1157ef1ca1105e63616fca58 WatchSource:0}: Error finding container 01d55d36e0dae09a8d1e2f6eb15f8d8b2ef7bc7c1157ef1ca1105e63616fca58: Status 404 returned error can't find the container with id 01d55d36e0dae09a8d1e2f6eb15f8d8b2ef7bc7c1157ef1ca1105e63616fca58
Sep 19 08:38:04.300262 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:04.300209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" event={"ID":"5b9ff759-ad0e-4d35-a354-e165df7b9152","Type":"ContainerStarted","Data":"3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"}
Sep 19 08:38:04.300643 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:04.300283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" event={"ID":"5b9ff759-ad0e-4d35-a354-e165df7b9152","Type":"ContainerStarted","Data":"01d55d36e0dae09a8d1e2f6eb15f8d8b2ef7bc7c1157ef1ca1105e63616fca58"}
Sep 19 08:38:04.300643 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:04.300309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:04.319492 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:04.319446    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" podStartSLOduration=1.319430029 podStartE2EDuration="1.319430029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:38:04.318247312 +0000 UTC m=+1036.504439271" watchObservedRunningTime="2026-09-19 08:38:04.319430029 +0000 UTC m=+1036.505621985"
Sep 19 08:38:22.705426 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:22.705392    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"]
Sep 19 08:38:31.951606 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:31.951565    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:35.307181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:35.307150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:37.011170 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:37.011136    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"]
Sep 19 08:38:37.724663 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:37.724626    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main" containerID="cri-o://7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39" gracePeriod=45
Sep 19 08:38:41.951457 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:41.951416    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:45.307494 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:45.307458    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:48.766084 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.766047    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 19 08:38:48.766084 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:38:48.766082    2580 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerName="main" containerID="cri-o://3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"
Sep 19 08:38:48.766474 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.766120    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerName="main" containerID="cri-o://3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd" gracePeriod=49
Sep 19 08:38:48.901842 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.901817    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh_5b9ff759-ad0e-4d35-a354-e165df7b9152/main/0.log"
Sep 19 08:38:48.904129 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.904109    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:48.966210 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.966176    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpd2p\" (UniqueName: \"kubernetes.io/projected/5b9ff759-ad0e-4d35-a354-e165df7b9152-kube-api-access-cpd2p\") pod \"5b9ff759-ad0e-4d35-a354-e165df7b9152\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") "
Sep 19 08:38:48.966210 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.966213    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9ff759-ad0e-4d35-a354-e165df7b9152-tls-certs\") pod \"5b9ff759-ad0e-4d35-a354-e165df7b9152\" (UID: \"5b9ff759-ad0e-4d35-a354-e165df7b9152\") "
Sep 19 08:38:48.968387 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.968363    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9ff759-ad0e-4d35-a354-e165df7b9152-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b9ff759-ad0e-4d35-a354-e165df7b9152" (UID: "5b9ff759-ad0e-4d35-a354-e165df7b9152"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:38:48.968479 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:48.968428    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9ff759-ad0e-4d35-a354-e165df7b9152-kube-api-access-cpd2p" (OuterVolumeSpecName: "kube-api-access-cpd2p") pod "5b9ff759-ad0e-4d35-a354-e165df7b9152" (UID: "5b9ff759-ad0e-4d35-a354-e165df7b9152"). InnerVolumeSpecName "kube-api-access-cpd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:38:49.067180 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.067109    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpd2p\" (UniqueName: \"kubernetes.io/projected/5b9ff759-ad0e-4d35-a354-e165df7b9152-kube-api-access-cpd2p\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:38:49.067180 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.067134    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b9ff759-ad0e-4d35-a354-e165df7b9152-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:38:49.487834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.487805    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh_5b9ff759-ad0e-4d35-a354-e165df7b9152/main/0.log"
Sep 19 08:38:49.487993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.487846    2580 generic.go:356] "Generic (PLEG): container finished" podID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerID="3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd" exitCode=1
Sep 19 08:38:49.487993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.487878    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" event={"ID":"5b9ff759-ad0e-4d35-a354-e165df7b9152","Type":"ContainerDied","Data":"3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"}
Sep 19 08:38:49.487993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.487925    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"
Sep 19 08:38:49.487993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.487968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh" event={"ID":"5b9ff759-ad0e-4d35-a354-e165df7b9152","Type":"ContainerDied","Data":"01d55d36e0dae09a8d1e2f6eb15f8d8b2ef7bc7c1157ef1ca1105e63616fca58"}
Sep 19 08:38:49.487993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.487992    2580 scope.go:117] "RemoveContainer" containerID="3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"
Sep 19 08:38:49.496710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.496695    2580 scope.go:117] "RemoveContainer" containerID="3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"
Sep 19 08:38:49.496985 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:38:49.496959    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd\": container with ID starting with 3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd not found: ID does not exist" containerID="3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"
Sep 19 08:38:49.497032 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.496993    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd"} err="failed to get container status \"3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd\": rpc error: code = NotFound desc = could not find container \"3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd\": container with ID starting with 3314e03b17615c95b395fe495d3ff636fb6db3af5385784385578b3694e53cfd not found: ID does not exist"
Sep 19 08:38:49.508302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.508279    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"]
Sep 19 08:38:49.514250 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:49.514213    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5kt4wh"]
Sep 19 08:38:50.080001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:50.079971    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" path="/var/lib/kubelet/pods/5b9ff759-ad0e-4d35-a354-e165df7b9152/volumes"
Sep 19 08:38:51.951812 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:51.951779    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:38:51.952156 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:38:51.951872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:39:01.951308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:01.951271    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:39:07.969641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:07.969616    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:39:08.019183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.019103    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-tls-certs\") pod \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") "
Sep 19 08:39:08.019349 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.019204    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb26f\" (UniqueName: \"kubernetes.io/projected/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-kube-api-access-qb26f\") pod \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\" (UID: \"f584e61e-bf1a-46b1-afa3-b7d426ee77ad\") "
Sep 19 08:39:08.021320 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.021295    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f584e61e-bf1a-46b1-afa3-b7d426ee77ad" (UID: "f584e61e-bf1a-46b1-afa3-b7d426ee77ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:39:08.021418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.021403    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-kube-api-access-qb26f" (OuterVolumeSpecName: "kube-api-access-qb26f") pod "f584e61e-bf1a-46b1-afa3-b7d426ee77ad" (UID: "f584e61e-bf1a-46b1-afa3-b7d426ee77ad"). InnerVolumeSpecName "kube-api-access-qb26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:39:08.120191 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.120165    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb26f\" (UniqueName: \"kubernetes.io/projected/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-kube-api-access-qb26f\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:39:08.120191 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.120191    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f584e61e-bf1a-46b1-afa3-b7d426ee77ad-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:39:08.563834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.563797    2580 generic.go:356] "Generic (PLEG): container finished" podID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerID="7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39" exitCode=0
Sep 19 08:39:08.563995 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.563843    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"
Sep 19 08:39:08.563995 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.563882    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" event={"ID":"f584e61e-bf1a-46b1-afa3-b7d426ee77ad","Type":"ContainerDied","Data":"7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39"}
Sep 19 08:39:08.563995 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.563948    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb" event={"ID":"f584e61e-bf1a-46b1-afa3-b7d426ee77ad","Type":"ContainerDied","Data":"4367b52f9c5cbf1a8a26fced59b9a9e037b70658bc75fc9493416c147a3ec541"}
Sep 19 08:39:08.563995 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.563984    2580 scope.go:117] "RemoveContainer" containerID="7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39"
Sep 19 08:39:08.572259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.572240    2580 scope.go:117] "RemoveContainer" containerID="7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39"
Sep 19 08:39:08.572524 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:39:08.572507    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39\": container with ID starting with 7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39 not found: ID does not exist" containerID="7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39"
Sep 19 08:39:08.572568 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.572533    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39"} err="failed to get container status \"7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39\": rpc error: code = NotFound desc = could not find container \"7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39\": container with ID starting with 7cb1167e6fd063b621bd4d5287cabf70d47417eb94e59c734581b8fcf5457d39 not found: ID does not exist"
Sep 19 08:39:08.579421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.579399    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"]
Sep 19 08:39:08.585311 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:08.585290    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-schepgmvb"]
Sep 19 08:39:10.078764 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:10.078734    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" path="/var/lib/kubelet/pods/f584e61e-bf1a-46b1-afa3-b7d426ee77ad/volumes"
Sep 19 08:39:18.366908 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.366871    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"]
Sep 19 08:39:18.367371 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.367351    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main"
Sep 19 08:39:18.367420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.367375    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main"
Sep 19 08:39:18.367420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.367390    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerName="main"
Sep 19 08:39:18.367420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.367399    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerName="main"
Sep 19 08:39:18.367519 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.367498    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f584e61e-bf1a-46b1-afa3-b7d426ee77ad" containerName="main"
Sep 19 08:39:18.367519 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.367511    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9ff759-ad0e-4d35-a354-e165df7b9152" containerName="main"
Sep 19 08:39:18.372154 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.372126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.374729 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.374707    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 19 08:39:18.375658 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.375638    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 19 08:39:18.376016 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.375995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-r94v6\""
Sep 19 08:39:18.376129 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.376038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 19 08:39:18.378209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.378190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"]
Sep 19 08:39:18.504610 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.504574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.504610 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.504610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.504814 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.504638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-home\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.504814 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.504718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xxqx\" (UniqueName: \"kubernetes.io/projected/d90aa123-29ea-4f70-a4de-264f4bceb587-kube-api-access-2xxqx\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.504814 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.504755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90aa123-29ea-4f70-a4de-264f4bceb587-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.504913 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.504821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605695 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-home\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605695 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605545    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xxqx\" (UniqueName: \"kubernetes.io/projected/d90aa123-29ea-4f70-a4de-264f4bceb587-kube-api-access-2xxqx\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605695 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90aa123-29ea-4f70-a4de-264f4bceb587-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605695 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.605978 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.605932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-home\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.606034 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.606015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.607966 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.607948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-dshm\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.608137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.608121    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90aa123-29ea-4f70-a4de-264f4bceb587-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.613052 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.613034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xxqx\" (UniqueName: \"kubernetes.io/projected/d90aa123-29ea-4f70-a4de-264f4bceb587-kube-api-access-2xxqx\") pod \"fc-smoke-test-kserve-7f4687897-vpxp5\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.684002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.683922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:18.749804 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.749768    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"]
Sep 19 08:39:18.753636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.753613    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:18.756070 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.755995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-87hv4\""
Sep 19 08:39:18.763399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.763373    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"]
Sep 19 08:39:18.908021 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.907984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqsl\" (UniqueName: \"kubernetes.io/projected/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-kube-api-access-nrqsl\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:18.908021 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.908023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:18.954107 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:18.954084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"]
Sep 19 08:39:18.955944 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:39:18.955912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90aa123_29ea_4f70_a4de_264f4bceb587.slice/crio-2a278b88ecab2f2421ca4281a5925caafafd86847b7f618ab6961652de50a065 WatchSource:0}: Error finding container 2a278b88ecab2f2421ca4281a5925caafafd86847b7f618ab6961652de50a065: Status 404 returned error can't find the container with id 2a278b88ecab2f2421ca4281a5925caafafd86847b7f618ab6961652de50a065
Sep 19 08:39:19.009291 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.009269    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqsl\" (UniqueName: \"kubernetes.io/projected/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-kube-api-access-nrqsl\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:19.009386 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.009299    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:19.011523 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.011500    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:19.016589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.016569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqsl\" (UniqueName: \"kubernetes.io/projected/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-kube-api-access-nrqsl\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:19.065377 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.065349    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:19.334856 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.334776    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"]
Sep 19 08:39:19.337593 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:39:19.337563    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc5d33c_7619_4a5a_a8b2_82de70d1b4e1.slice/crio-43df04d572b3f65e015bab8c080b02376b591a1e6b6e7c629b6159500c0b859f WatchSource:0}: Error finding container 43df04d572b3f65e015bab8c080b02376b591a1e6b6e7c629b6159500c0b859f: Status 404 returned error can't find the container with id 43df04d572b3f65e015bab8c080b02376b591a1e6b6e7c629b6159500c0b859f
Sep 19 08:39:19.617576 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.617485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" event={"ID":"d90aa123-29ea-4f70-a4de-264f4bceb587","Type":"ContainerStarted","Data":"cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b"}
Sep 19 08:39:19.617576 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.617572    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" event={"ID":"d90aa123-29ea-4f70-a4de-264f4bceb587","Type":"ContainerStarted","Data":"2a278b88ecab2f2421ca4281a5925caafafd86847b7f618ab6961652de50a065"}
Sep 19 08:39:19.619084 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.619060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" event={"ID":"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1","Type":"ContainerStarted","Data":"f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333"}
Sep 19 08:39:19.619236 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.619097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" event={"ID":"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1","Type":"ContainerStarted","Data":"43df04d572b3f65e015bab8c080b02376b591a1e6b6e7c629b6159500c0b859f"}
Sep 19 08:39:19.619341 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.619209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:39:19.633110 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.633071    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" podStartSLOduration=1.633057216 podStartE2EDuration="1.633057216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:39:19.632773587 +0000 UTC m=+1111.818965545" watchObservedRunningTime="2026-09-19 08:39:19.633057216 +0000 UTC m=+1111.819249173"
Sep 19 08:39:19.649323 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.649276    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" podStartSLOduration=1.649262191 podStartE2EDuration="1.649262191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:39:19.647780092 +0000 UTC m=+1111.833972062" watchObservedRunningTime="2026-09-19 08:39:19.649262191 +0000 UTC m=+1111.835454156"
Sep 19 08:39:19.684517 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:19.684490    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:28.684882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:28.684850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:28.697149 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:28.697129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:29.675555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:29.675522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:39:50.625720 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:39:50.625688    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:40:13.600860 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.600764    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"]
Sep 19 08:40:13.602773 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.602745    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"]
Sep 19 08:40:13.611395 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:40:13.611358    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:40:13.611395 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:40:13.611395 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:40:13.611395 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b" cmd=["/bin/sleep","15"]
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.611420    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" message="" err=<
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:40:13.611437    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerName="main" containerID="cri-o://cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b"
Sep 19 08:40:13.611661 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.611477    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerName="main" containerID="cri-o://cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b" gracePeriod=60
Sep 19 08:40:13.665193 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.665157    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused"
Sep 19 08:40:13.860236 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.860147    2580 generic.go:356] "Generic (PLEG): container finished" podID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerID="cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b" exitCode=0
Sep 19 08:40:13.860236 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.860198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" event={"ID":"d90aa123-29ea-4f70-a4de-264f4bceb587","Type":"ContainerDied","Data":"cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b"}
Sep 19 08:40:13.957988 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:13.957962    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:40:14.067059 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-home\") pod \"d90aa123-29ea-4f70-a4de-264f4bceb587\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") "
Sep 19 08:40:14.067278 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90aa123-29ea-4f70-a4de-264f4bceb587-tls-certs\") pod \"d90aa123-29ea-4f70-a4de-264f4bceb587\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") "
Sep 19 08:40:14.067278 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067089    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xxqx\" (UniqueName: \"kubernetes.io/projected/d90aa123-29ea-4f70-a4de-264f4bceb587-kube-api-access-2xxqx\") pod \"d90aa123-29ea-4f70-a4de-264f4bceb587\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") "
Sep 19 08:40:14.067278 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-tmp-dir\") pod \"d90aa123-29ea-4f70-a4de-264f4bceb587\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") "
Sep 19 08:40:14.067278 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-model-cache\") pod \"d90aa123-29ea-4f70-a4de-264f4bceb587\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") "
Sep 19 08:40:14.067278 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067249    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-dshm\") pod \"d90aa123-29ea-4f70-a4de-264f4bceb587\" (UID: \"d90aa123-29ea-4f70-a4de-264f4bceb587\") "
Sep 19 08:40:14.067524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067429    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-home" (OuterVolumeSpecName: "home") pod "d90aa123-29ea-4f70-a4de-264f4bceb587" (UID: "d90aa123-29ea-4f70-a4de-264f4bceb587"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:14.067524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d90aa123-29ea-4f70-a4de-264f4bceb587" (UID: "d90aa123-29ea-4f70-a4de-264f4bceb587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:14.067595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.067516    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-model-cache" (OuterVolumeSpecName: "model-cache") pod "d90aa123-29ea-4f70-a4de-264f4bceb587" (UID: "d90aa123-29ea-4f70-a4de-264f4bceb587"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:14.069412 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.069389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90aa123-29ea-4f70-a4de-264f4bceb587-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d90aa123-29ea-4f70-a4de-264f4bceb587" (UID: "d90aa123-29ea-4f70-a4de-264f4bceb587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:40:14.069488 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.069449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-dshm" (OuterVolumeSpecName: "dshm") pod "d90aa123-29ea-4f70-a4de-264f4bceb587" (UID: "d90aa123-29ea-4f70-a4de-264f4bceb587"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:40:14.069524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.069483    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90aa123-29ea-4f70-a4de-264f4bceb587-kube-api-access-2xxqx" (OuterVolumeSpecName: "kube-api-access-2xxqx") pod "d90aa123-29ea-4f70-a4de-264f4bceb587" (UID: "d90aa123-29ea-4f70-a4de-264f4bceb587"). InnerVolumeSpecName "kube-api-access-2xxqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:40:14.168559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.168478    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:14.168559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.168503    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d90aa123-29ea-4f70-a4de-264f4bceb587-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:14.168559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.168513    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xxqx\" (UniqueName: \"kubernetes.io/projected/d90aa123-29ea-4f70-a4de-264f4bceb587-kube-api-access-2xxqx\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:14.168559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.168523    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:14.168559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.168532    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:14.168559 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.168540    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d90aa123-29ea-4f70-a4de-264f4bceb587-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:14.865649 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.865622    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"
Sep 19 08:40:14.866050 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.865622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5" event={"ID":"d90aa123-29ea-4f70-a4de-264f4bceb587","Type":"ContainerDied","Data":"2a278b88ecab2f2421ca4281a5925caafafd86847b7f618ab6961652de50a065"}
Sep 19 08:40:14.866050 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.865760    2580 scope.go:117] "RemoveContainer" containerID="cb4ed3a327c8b293f614c7326b847ab3ccdd7e1d971a80447bd7bbfd67b3265b"
Sep 19 08:40:14.880326 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.880302    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"]
Sep 19 08:40:14.884026 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:14.883993    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-vpxp5"]
Sep 19 08:40:16.079116 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:16.079082    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" path="/var/lib/kubelet/pods/d90aa123-29ea-4f70-a4de-264f4bceb587/volumes"
Sep 19 08:40:20.626146 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:20.626097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:40:28.621840 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:28.621803    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main" containerID="cri-o://f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333" gracePeriod=45
Sep 19 08:40:30.625868 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:30.625824    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:40:40.625410 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:40.625370    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:40:40.625859 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:40.625481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:40:50.626067 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:50.626025    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:40:58.862985 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:58.862958    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:40:58.943737 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:58.943710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-tls-certs\") pod \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") "
Sep 19 08:40:58.943880 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:58.943778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrqsl\" (UniqueName: \"kubernetes.io/projected/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-kube-api-access-nrqsl\") pod \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\" (UID: \"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1\") "
Sep 19 08:40:58.945815 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:58.945787    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" (UID: "efc5d33c-7619-4a5a-a8b2-82de70d1b4e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:40:58.945934 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:58.945862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-kube-api-access-nrqsl" (OuterVolumeSpecName: "kube-api-access-nrqsl") pod "efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" (UID: "efc5d33c-7619-4a5a-a8b2-82de70d1b4e1"). InnerVolumeSpecName "kube-api-access-nrqsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:40:59.044354 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.044323    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:59.044354 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.044347    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqsl\" (UniqueName: \"kubernetes.io/projected/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1-kube-api-access-nrqsl\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:40:59.057769 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.057740    2580 generic.go:356] "Generic (PLEG): container finished" podID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerID="f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333" exitCode=0
Sep 19 08:40:59.057888 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.057790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"
Sep 19 08:40:59.057888 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.057802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" event={"ID":"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1","Type":"ContainerDied","Data":"f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333"}
Sep 19 08:40:59.057888 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.057852    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl" event={"ID":"efc5d33c-7619-4a5a-a8b2-82de70d1b4e1","Type":"ContainerDied","Data":"43df04d572b3f65e015bab8c080b02376b591a1e6b6e7c629b6159500c0b859f"}
Sep 19 08:40:59.057888 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.057883    2580 scope.go:117] "RemoveContainer" containerID="f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333"
Sep 19 08:40:59.066468 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.066450    2580 scope.go:117] "RemoveContainer" containerID="f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333"
Sep 19 08:40:59.066692 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:40:59.066674    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333\": container with ID starting with f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333 not found: ID does not exist" containerID="f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333"
Sep 19 08:40:59.066738 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.066699    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333"} err="failed to get container status \"f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333\": rpc error: code = NotFound desc = could not find container \"f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333\": container with ID starting with f981b54670779e9bf2acc6f13f5da0c9cef10c0261c93b4f4a56c7f38e183333 not found: ID does not exist"
Sep 19 08:40:59.077631 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.077611    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"]
Sep 19 08:40:59.081710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:40:59.081691    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-2cnkl"]
Sep 19 08:41:00.078130 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:00.078088    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" path="/var/lib/kubelet/pods/efc5d33c-7619-4a5a-a8b2-82de70d1b4e1/volumes"
Sep 19 08:41:20.389711 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.389675    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"]
Sep 19 08:41:20.390174 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.390161    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerName="main"
Sep 19 08:41:20.390263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.390180    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerName="main"
Sep 19 08:41:20.390263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.390200    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main"
Sep 19 08:41:20.390263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.390208    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main"
Sep 19 08:41:20.390419 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.390305    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90aa123-29ea-4f70-a4de-264f4bceb587" containerName="main"
Sep 19 08:41:20.390419 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.390320    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc5d33c-7619-4a5a-a8b2-82de70d1b4e1" containerName="main"
Sep 19 08:41:20.393168 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.393145    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.396826 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.396793    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 19 08:41:20.396967 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.396853    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-442t7\""
Sep 19 08:41:20.396967 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.396800    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 19 08:41:20.396967 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.396857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 19 08:41:20.396967 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.396800    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-xjmgp\""
Sep 19 08:41:20.403512 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.403489    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"]
Sep 19 08:41:20.516259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.516191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9lz\" (UniqueName: \"kubernetes.io/projected/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-kube-api-access-8n9lz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.516460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.516310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.617553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.617516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.617726 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.617591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9lz\" (UniqueName: \"kubernetes.io/projected/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-kube-api-access-8n9lz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.620069 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.620043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.625589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.625562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9lz\" (UniqueName: \"kubernetes.io/projected/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-kube-api-access-8n9lz\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.704545 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.704509    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:20.973947 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:20.973905    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"]
Sep 19 08:41:20.977484 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:41:20.977452    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1751cbd_c82f_46b0_b1ca_5ba27a1e7697.slice/crio-f5dd1bc1a780b79eb1b3411728d7e564dd74d66984d6d7f16d1ccb8bd0253370 WatchSource:0}: Error finding container f5dd1bc1a780b79eb1b3411728d7e564dd74d66984d6d7f16d1ccb8bd0253370: Status 404 returned error can't find the container with id f5dd1bc1a780b79eb1b3411728d7e564dd74d66984d6d7f16d1ccb8bd0253370
Sep 19 08:41:21.145379 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:21.145338    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" event={"ID":"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697","Type":"ContainerStarted","Data":"713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06"}
Sep 19 08:41:21.145569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:21.145403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" event={"ID":"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697","Type":"ContainerStarted","Data":"f5dd1bc1a780b79eb1b3411728d7e564dd74d66984d6d7f16d1ccb8bd0253370"}
Sep 19 08:41:21.145569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:21.145533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:21.164980 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:21.164920    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podStartSLOduration=1.164904449 podStartE2EDuration="1.164904449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:21.163600405 +0000 UTC m=+1233.349792354" watchObservedRunningTime="2026-09-19 08:41:21.164904449 +0000 UTC m=+1233.351096406"
Sep 19 08:41:44.098064 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.098029    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"]
Sep 19 08:41:44.102018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.101992    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.110325 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.110305    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-7khck\""
Sep 19 08:41:44.110736 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.110714    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-c787g\""
Sep 19 08:41:44.110836 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.110777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 19 08:41:44.111031 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.111010    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 19 08:41:44.116333 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.116312    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 19 08:41:44.136637 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.136610    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"]
Sep 19 08:41:44.211450 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.211414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sc4d\" (UniqueName: \"kubernetes.io/projected/a04ca9cb-f019-4627-b342-aeedb70d1cc9-kube-api-access-8sc4d\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.211624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.211457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.211624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.211482    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.211624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.211575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.211624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.211601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.211818 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.211629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.312529 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.312718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sc4d\" (UniqueName: \"kubernetes.io/projected/a04ca9cb-f019-4627-b342-aeedb70d1cc9-kube-api-access-8sc4d\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.312718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.312718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.312718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.312718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312658    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.313002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.313068 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.312989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.313122 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.313070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.314997 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.314968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.315196 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.315182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.323260 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.323240    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sc4d\" (UniqueName: \"kubernetes.io/projected/a04ca9cb-f019-4627-b342-aeedb70d1cc9-kube-api-access-8sc4d\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.412522 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.412420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:44.438719 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.438690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"]
Sep 19 08:41:44.442265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.442235    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.444745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.444723    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-tlnjd\""
Sep 19 08:41:44.453119 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.452994    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"]
Sep 19 08:41:44.514813 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.514785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.514959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.514820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf558\" (UniqueName: \"kubernetes.io/projected/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-kube-api-access-jf558\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.615672 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.615635    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf558\" (UniqueName: \"kubernetes.io/projected/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-kube-api-access-jf558\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.615838 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.615759    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.618189 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.618163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.623644 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.623626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf558\" (UniqueName: \"kubernetes.io/projected/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-kube-api-access-jf558\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:44.692481 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.692453    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"]
Sep 19 08:41:44.694717 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:41:44.694690    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04ca9cb_f019_4627_b342_aeedb70d1cc9.slice/crio-809b7a9c5bee0c00a91ecdae0086c24ca32934c2e040e36cb24fd822cda7efb9 WatchSource:0}: Error finding container 809b7a9c5bee0c00a91ecdae0086c24ca32934c2e040e36cb24fd822cda7efb9: Status 404 returned error can't find the container with id 809b7a9c5bee0c00a91ecdae0086c24ca32934c2e040e36cb24fd822cda7efb9
Sep 19 08:41:44.696701 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.696679    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:41:44.756166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:44.756135    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:45.024481 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:45.024289    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"]
Sep 19 08:41:45.027183 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:41:45.027154    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b27dfa_dc80_4ef6_bbc4_b3a577de2d25.slice/crio-0b4f07d55d23e08683211608572761c5ffa10af2011db0421b4bf65515645dc9 WatchSource:0}: Error finding container 0b4f07d55d23e08683211608572761c5ffa10af2011db0421b4bf65515645dc9: Status 404 returned error can't find the container with id 0b4f07d55d23e08683211608572761c5ffa10af2011db0421b4bf65515645dc9
Sep 19 08:41:45.246369 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:45.246333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" event={"ID":"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25","Type":"ContainerStarted","Data":"4b5490b479af9d1d63e1810d190674253989f8f30c1fc1464b96c0935832d061"}
Sep 19 08:41:45.246830 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:45.246392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" event={"ID":"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25","Type":"ContainerStarted","Data":"0b4f07d55d23e08683211608572761c5ffa10af2011db0421b4bf65515645dc9"}
Sep 19 08:41:45.246830 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:45.246443    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:41:45.247663 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:45.247641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" event={"ID":"a04ca9cb-f019-4627-b342-aeedb70d1cc9","Type":"ContainerStarted","Data":"809b7a9c5bee0c00a91ecdae0086c24ca32934c2e040e36cb24fd822cda7efb9"}
Sep 19 08:41:45.266270 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:45.266193    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podStartSLOduration=1.266179162 podStartE2EDuration="1.266179162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:45.264738651 +0000 UTC m=+1257.450930610" watchObservedRunningTime="2026-09-19 08:41:45.266179162 +0000 UTC m=+1257.452371118"
Sep 19 08:41:46.253579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:46.253547    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" event={"ID":"a04ca9cb-f019-4627-b342-aeedb70d1cc9","Type":"ContainerStarted","Data":"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"}
Sep 19 08:41:47.260245 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:47.260194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" event={"ID":"a04ca9cb-f019-4627-b342-aeedb70d1cc9","Type":"ContainerStarted","Data":"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"}
Sep 19 08:41:47.260617 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:47.260330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:47.279598 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:47.279537    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" podStartSLOduration=1.928015874 podStartE2EDuration="3.279519961s" totalImagesPullingTime="1.351504087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:47.277866844 +0000 UTC m=+1259.464058814" watchObservedRunningTime="2026-09-19 08:41:47.279519961 +0000 UTC m=+1259.465711919"
Sep 19 08:41:47.413129 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:47.413090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:52.152926 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:52.152898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:41:54.413056 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:54.413023    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:54.425286 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:54.425260    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:55.306383 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:55.306355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:41:56.311837 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:41:56.311809    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:42:00.994877 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:00.994842    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"]
Sep 19 08:42:02.152905 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:02.152861    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:12.152059 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:12.152013    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:16.014161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:16.014119    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main" containerID="cri-o://713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06" gracePeriod=45
Sep 19 08:42:16.255912 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:16.255882    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:42:22.153115 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.153049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:22.153607 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.153300    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:42:22.425762 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.425683    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"]
Sep 19 08:42:22.440273 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.440204    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"]
Sep 19 08:42:22.448881 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:42:22.448839    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:42:22.448881 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:22.448881 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:22.448881 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7" cmd=["/bin/sleep","15"]
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.448902    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" message="" err=<
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:42:22.448919    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="main" containerID="cri-o://36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"
Sep 19 08:42:22.449194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.449011    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="main" containerID="cri-o://36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7" gracePeriod=60
Sep 19 08:42:22.662381 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.662348    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="llm-d-routing-sidecar" containerID="cri-o://600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e" gracePeriod=60
Sep 19 08:42:22.911141 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:22.911113    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:42:23.042340 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sc4d\" (UniqueName: \"kubernetes.io/projected/a04ca9cb-f019-4627-b342-aeedb70d1cc9-kube-api-access-8sc4d\") pod \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") "
Sep 19 08:42:23.042532 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042366    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-home\") pod \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") "
Sep 19 08:42:23.042532 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042390    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-model-cache\") pod \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") "
Sep 19 08:42:23.042532 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042433    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tmp-dir\") pod \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") "
Sep 19 08:42:23.042532 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-dshm\") pod \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") "
Sep 19 08:42:23.042532 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042529    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tls-certs\") pod \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\" (UID: \"a04ca9cb-f019-4627-b342-aeedb70d1cc9\") "
Sep 19 08:42:23.042799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-home" (OuterVolumeSpecName: "home") pod "a04ca9cb-f019-4627-b342-aeedb70d1cc9" (UID: "a04ca9cb-f019-4627-b342-aeedb70d1cc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:23.042799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a04ca9cb-f019-4627-b342-aeedb70d1cc9" (UID: "a04ca9cb-f019-4627-b342-aeedb70d1cc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:23.042799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a04ca9cb-f019-4627-b342-aeedb70d1cc9" (UID: "a04ca9cb-f019-4627-b342-aeedb70d1cc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:23.042924 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042818    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.042924 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.042836    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.044733 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.044711    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a04ca9cb-f019-4627-b342-aeedb70d1cc9" (UID: "a04ca9cb-f019-4627-b342-aeedb70d1cc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:42:23.044910 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.044891    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-dshm" (OuterVolumeSpecName: "dshm") pod "a04ca9cb-f019-4627-b342-aeedb70d1cc9" (UID: "a04ca9cb-f019-4627-b342-aeedb70d1cc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:42:23.044988 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.044906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a04ca9cb-f019-4627-b342-aeedb70d1cc9-kube-api-access-8sc4d" (OuterVolumeSpecName: "kube-api-access-8sc4d") pod "a04ca9cb-f019-4627-b342-aeedb70d1cc9" (UID: "a04ca9cb-f019-4627-b342-aeedb70d1cc9"). InnerVolumeSpecName "kube-api-access-8sc4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:42:23.143238 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.143183    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.143238 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.143211    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a04ca9cb-f019-4627-b342-aeedb70d1cc9-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.143238 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.143244    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a04ca9cb-f019-4627-b342-aeedb70d1cc9-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.143448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.143256    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sc4d\" (UniqueName: \"kubernetes.io/projected/a04ca9cb-f019-4627-b342-aeedb70d1cc9-kube-api-access-8sc4d\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:23.417445 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417356    2580 generic.go:356] "Generic (PLEG): container finished" podID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerID="36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7" exitCode=0
Sep 19 08:42:23.417445 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417381    2580 generic.go:356] "Generic (PLEG): container finished" podID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerID="600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e" exitCode=0
Sep 19 08:42:23.417445 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417411    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"
Sep 19 08:42:23.417943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" event={"ID":"a04ca9cb-f019-4627-b342-aeedb70d1cc9","Type":"ContainerDied","Data":"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"}
Sep 19 08:42:23.417943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417498    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" event={"ID":"a04ca9cb-f019-4627-b342-aeedb70d1cc9","Type":"ContainerDied","Data":"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"}
Sep 19 08:42:23.417943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j" event={"ID":"a04ca9cb-f019-4627-b342-aeedb70d1cc9","Type":"ContainerDied","Data":"809b7a9c5bee0c00a91ecdae0086c24ca32934c2e040e36cb24fd822cda7efb9"}
Sep 19 08:42:23.417943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.417537    2580 scope.go:117] "RemoveContainer" containerID="36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"
Sep 19 08:42:23.426875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.426861    2580 scope.go:117] "RemoveContainer" containerID="600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"
Sep 19 08:42:23.434579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.434563    2580 scope.go:117] "RemoveContainer" containerID="36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"
Sep 19 08:42:23.434847 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:42:23.434827    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7\": container with ID starting with 36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7 not found: ID does not exist" containerID="36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"
Sep 19 08:42:23.434893 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.434855    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"} err="failed to get container status \"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7\": rpc error: code = NotFound desc = could not find container \"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7\": container with ID starting with 36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7 not found: ID does not exist"
Sep 19 08:42:23.434893 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.434872    2580 scope.go:117] "RemoveContainer" containerID="600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"
Sep 19 08:42:23.435128 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:42:23.435108    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e\": container with ID starting with 600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e not found: ID does not exist" containerID="600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"
Sep 19 08:42:23.435182 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.435134    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"} err="failed to get container status \"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e\": rpc error: code = NotFound desc = could not find container \"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e\": container with ID starting with 600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e not found: ID does not exist"
Sep 19 08:42:23.435182 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.435150    2580 scope.go:117] "RemoveContainer" containerID="36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"
Sep 19 08:42:23.435390 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.435373    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7"} err="failed to get container status \"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7\": rpc error: code = NotFound desc = could not find container \"36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7\": container with ID starting with 36c95c84e7eb0ef0bdd2fe73781d5011421863790dee2b48c4911bacff9e50d7 not found: ID does not exist"
Sep 19 08:42:23.435435 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.435390    2580 scope.go:117] "RemoveContainer" containerID="600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"
Sep 19 08:42:23.435583 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.435568    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e"} err="failed to get container status \"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e\": rpc error: code = NotFound desc = could not find container \"600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e\": container with ID starting with 600abb482ed36ee46501ea5f5dd52cceabece07635935ee18828975f0c79880e not found: ID does not exist"
Sep 19 08:42:23.438802 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.438781    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"]
Sep 19 08:42:23.444657 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:23.444637    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-fl87j"]
Sep 19 08:42:24.079781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:24.079743    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" path="/var/lib/kubelet/pods/a04ca9cb-f019-4627-b342-aeedb70d1cc9/volumes"
Sep 19 08:42:26.256099 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:26.256059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:32.152372 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:32.152333    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:36.255963 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:36.255926    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:37.445184 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:37.445128    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main" containerID="cri-o://4b5490b479af9d1d63e1810d190674253989f8f30c1fc1464b96c0935832d061" gracePeriod=45
Sep 19 08:42:42.152232 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:42.152191    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:46.255199 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.255158    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:42:46.255611 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.255302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:42:46.259809 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.259788    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:42:46.339064 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.338998    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-tls-certs\") pod \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") "
Sep 19 08:42:46.339195 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.339085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n9lz\" (UniqueName: \"kubernetes.io/projected/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-kube-api-access-8n9lz\") pod \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\" (UID: \"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697\") "
Sep 19 08:42:46.341211 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.341186    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-kube-api-access-8n9lz" (OuterVolumeSpecName: "kube-api-access-8n9lz") pod "d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" (UID: "d1751cbd-c82f-46b0-b1ca-5ba27a1e7697"). InnerVolumeSpecName "kube-api-access-8n9lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:42:46.341211 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.341201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" (UID: "d1751cbd-c82f-46b0-b1ca-5ba27a1e7697"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:42:46.440546 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.440516    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:46.440546 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.440541    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9lz\" (UniqueName: \"kubernetes.io/projected/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697-kube-api-access-8n9lz\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:42:46.514943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.514911    2580 generic.go:356] "Generic (PLEG): container finished" podID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerID="713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06" exitCode=0
Sep 19 08:42:46.515111 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.514963    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"
Sep 19 08:42:46.515111 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.514988    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" event={"ID":"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697","Type":"ContainerDied","Data":"713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06"}
Sep 19 08:42:46.515111 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.515027    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h" event={"ID":"d1751cbd-c82f-46b0-b1ca-5ba27a1e7697","Type":"ContainerDied","Data":"f5dd1bc1a780b79eb1b3411728d7e564dd74d66984d6d7f16d1ccb8bd0253370"}
Sep 19 08:42:46.515111 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.515048    2580 scope.go:117] "RemoveContainer" containerID="713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06"
Sep 19 08:42:46.524340 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.524324    2580 scope.go:117] "RemoveContainer" containerID="713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06"
Sep 19 08:42:46.524638 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:42:46.524614    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06\": container with ID starting with 713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06 not found: ID does not exist" containerID="713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06"
Sep 19 08:42:46.524731 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.524643    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06"} err="failed to get container status \"713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06\": rpc error: code = NotFound desc = could not find container \"713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06\": container with ID starting with 713662a915a7ccfed674c5020a4fa6eda9472f516b19a194bd5fd45e81f9de06 not found: ID does not exist"
Sep 19 08:42:46.535976 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.535953    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"]
Sep 19 08:42:46.540315 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:46.540293    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-g7z6h"]
Sep 19 08:42:48.078718 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:48.078682    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" path="/var/lib/kubelet/pods/d1751cbd-c82f-46b0-b1ca-5ba27a1e7697/volumes"
Sep 19 08:42:56.255291 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:42:56.255246    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:43:06.255781 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:06.255694    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:43:07.601580 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.601547    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerID="4b5490b479af9d1d63e1810d190674253989f8f30c1fc1464b96c0935832d061" exitCode=0
Sep 19 08:43:07.601949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.601624    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" event={"ID":"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25","Type":"ContainerDied","Data":"4b5490b479af9d1d63e1810d190674253989f8f30c1fc1464b96c0935832d061"}
Sep 19 08:43:07.685256 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.685213    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:43:07.815281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.815169    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf558\" (UniqueName: \"kubernetes.io/projected/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-kube-api-access-jf558\") pod \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") "
Sep 19 08:43:07.815281 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.815212    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-tls-certs\") pod \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\" (UID: \"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25\") "
Sep 19 08:43:07.817203 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.817172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-kube-api-access-jf558" (OuterVolumeSpecName: "kube-api-access-jf558") pod "f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" (UID: "f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25"). InnerVolumeSpecName "kube-api-access-jf558". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:07.817314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.817207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" (UID: "f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:07.916557 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.916522    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf558\" (UniqueName: \"kubernetes.io/projected/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-kube-api-access-jf558\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:07.916557 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:07.916550    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:08.607414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:08.607375    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn" event={"ID":"f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25","Type":"ContainerDied","Data":"0b4f07d55d23e08683211608572761c5ffa10af2011db0421b4bf65515645dc9"}
Sep 19 08:43:08.607414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:08.607395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"
Sep 19 08:43:08.607909 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:08.607461    2580 scope.go:117] "RemoveContainer" containerID="4b5490b479af9d1d63e1810d190674253989f8f30c1fc1464b96c0935832d061"
Sep 19 08:43:08.627592 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:08.627569    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"]
Sep 19 08:43:08.631117 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:08.631097    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75f8cvn"]
Sep 19 08:43:10.079523 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:10.079493    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" path="/var/lib/kubelet/pods/f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25/volumes"
Sep 19 08:43:44.163450 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163416    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"]
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163711    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163722    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163737    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163749    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="llm-d-routing-sidecar"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163756    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="llm-d-routing-sidecar"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163763    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163769    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163837    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1751cbd-c82f-46b0-b1ca-5ba27a1e7697" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163844    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163851    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5b27dfa-dc80-4ef6-bbc4-b3a577de2d25" containerName="main"
Sep 19 08:43:44.163882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.163855    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a04ca9cb-f019-4627-b342-aeedb70d1cc9" containerName="llm-d-routing-sidecar"
Sep 19 08:43:44.166587 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.166570    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.170090 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.170060    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 19 08:43:44.170090 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.170076    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-zg8hf\""
Sep 19 08:43:44.170272 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.170153    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 19 08:43:44.170272 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.170149    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Sep 19 08:43:44.177649 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.177621    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"]
Sep 19 08:43:44.205261 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.205214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.205420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.205270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.205420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.205302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.205420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.205332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584bn\" (UniqueName: \"kubernetes.io/projected/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-kube-api-access-584bn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.205420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.205380    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.205569 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.205428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306020 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.305983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306020 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584bn\" (UniqueName: \"kubernetes.io/projected/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-kube-api-access-584bn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306075    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306101    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306137    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306542 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306542 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.306637 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.306568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.308466 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.308437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.308707 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.308689    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.315031 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.315007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-584bn\" (UniqueName: \"kubernetes.io/projected/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-kube-api-access-584bn\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.477276 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.477237    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:44.751835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:44.751812    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"]
Sep 19 08:43:44.754495 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:43:44.754464    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb9b1bd_cbb3_45ab_83e4_33c1ae15db5a.slice/crio-98c021ac6f865142f54be9dea04cb604af7063ef7fcb4b419e3749c3671f946f WatchSource:0}: Error finding container 98c021ac6f865142f54be9dea04cb604af7063ef7fcb4b419e3749c3671f946f: Status 404 returned error can't find the container with id 98c021ac6f865142f54be9dea04cb604af7063ef7fcb4b419e3749c3671f946f
Sep 19 08:43:45.758465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:45.758432    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" event={"ID":"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a","Type":"ContainerStarted","Data":"b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f"}
Sep 19 08:43:45.758853 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:45.758483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" event={"ID":"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a","Type":"ContainerStarted","Data":"98c021ac6f865142f54be9dea04cb604af7063ef7fcb4b419e3749c3671f946f"}
Sep 19 08:43:45.775724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:45.775675    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" podStartSLOduration=1.77565665 podStartE2EDuration="1.77565665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:45.773795731 +0000 UTC m=+1377.959987687" watchObservedRunningTime="2026-09-19 08:43:45.77565665 +0000 UTC m=+1377.961848673"
Sep 19 08:43:46.478127 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:46.478091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:51.576601 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.576559    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"]
Sep 19 08:43:51.585487 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:43:51.585454    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:43:51.585487 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:43:51.585487 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:43:51.585487 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f" cmd=["/bin/sleep","15"]
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.585504    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" message="" err=<
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:43:51.585517    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" podUID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" containerName="main" containerID="cri-o://b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f"
Sep 19 08:43:51.585712 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.585564    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" podUID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" containerName="main" containerID="cri-o://b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f" gracePeriod=60
Sep 19 08:43:51.786579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.786546    2580 generic.go:356] "Generic (PLEG): container finished" podID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" containerID="b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f" exitCode=0
Sep 19 08:43:51.786735 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.786594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" event={"ID":"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a","Type":"ContainerDied","Data":"b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f"}
Sep 19 08:43:51.833676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.833619    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:51.869496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-584bn\" (UniqueName: \"kubernetes.io/projected/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-kube-api-access-584bn\") pod \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") "
Sep 19 08:43:51.869646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869522    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tmp-dir\") pod \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") "
Sep 19 08:43:51.869646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869579    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-model-cache\") pod \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") "
Sep 19 08:43:51.869646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-dshm\") pod \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") "
Sep 19 08:43:51.869778 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tls-certs\") pod \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") "
Sep 19 08:43:51.869778 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869698    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-home\") pod \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\" (UID: \"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a\") "
Sep 19 08:43:51.869881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" (UID: "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:51.869881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869835    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" (UID: "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:51.869993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869926    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:51.869993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869948    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:51.869993 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.869980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-home" (OuterVolumeSpecName: "home") pod "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" (UID: "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:51.872402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.872362    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" (UID: "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:43:51.872668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.872643    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-dshm" (OuterVolumeSpecName: "dshm") pod "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" (UID: "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:43:51.872892 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.872858    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-kube-api-access-584bn" (OuterVolumeSpecName: "kube-api-access-584bn") pod "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" (UID: "dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a"). InnerVolumeSpecName "kube-api-access-584bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:43:51.970694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.970647    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:51.970694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.970688    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:51.970694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.970698    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-584bn\" (UniqueName: \"kubernetes.io/projected/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-kube-api-access-584bn\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:51.970694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:51.970708    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:43:52.791302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:52.791270    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"
Sep 19 08:43:52.791302 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:52.791279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn" event={"ID":"dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a","Type":"ContainerDied","Data":"98c021ac6f865142f54be9dea04cb604af7063ef7fcb4b419e3749c3671f946f"}
Sep 19 08:43:52.791841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:52.791351    2580 scope.go:117] "RemoveContainer" containerID="b2513d957ffb786c6934f7374c8a129f6f787edc6389c5dca0dc93005950863f"
Sep 19 08:43:52.809683 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:52.809651    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"]
Sep 19 08:43:52.811394 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:52.811368    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfvcmvn"]
Sep 19 08:43:54.079640 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:43:54.079560    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" path="/var/lib/kubelet/pods/dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a/volumes"
Sep 19 08:44:05.711126 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.711091    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"]
Sep 19 08:44:05.711599 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.711535    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" containerName="main"
Sep 19 08:44:05.711599 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.711552    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" containerName="main"
Sep 19 08:44:05.711673 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.711605    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb9b1bd-cbb3-45ab-83e4-33c1ae15db5a" containerName="main"
Sep 19 08:44:05.716050 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.716031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.719777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.719614    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 19 08:44:05.719777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.719636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 19 08:44:05.719777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.719623    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-f67wm\""
Sep 19 08:44:05.719777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.719664    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-n7zqj\""
Sep 19 08:44:05.719777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.719677    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 19 08:44:05.727773 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.727753    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"]
Sep 19 08:44:05.786347 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.786317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkfs\" (UniqueName: \"kubernetes.io/projected/59304542-9e02-46b6-ae52-7a07aa7bd940-kube-api-access-8pkfs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.786489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.786351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.786489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.786368    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.786489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.786416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.786489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.786471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59304542-9e02-46b6-ae52-7a07aa7bd940-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.786656 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.786494    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.887553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.887520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59304542-9e02-46b6-ae52-7a07aa7bd940-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.887553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.887553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.887780 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.887615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkfs\" (UniqueName: \"kubernetes.io/projected/59304542-9e02-46b6-ae52-7a07aa7bd940-kube-api-access-8pkfs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.887780 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.887638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.887780 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.887653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.887780 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.887669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.888119 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.888097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-home\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.888201 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.888122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.888201 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.888157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.889887 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.889867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.890150 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.890133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59304542-9e02-46b6-ae52-7a07aa7bd940-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.895183 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.895159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkfs\" (UniqueName: \"kubernetes.io/projected/59304542-9e02-46b6-ae52-7a07aa7bd940-kube-api-access-8pkfs\") pod \"scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:05.925677 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.925644    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"]
Sep 19 08:44:05.929534 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.929517    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:05.932015 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.931991    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-fjxbd\""
Sep 19 08:44:05.936413 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.936390    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"]
Sep 19 08:44:05.988518 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.988437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a5bfcd-89db-4f2f-8867-dcd39198adf9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk675f\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:05.988518 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:05.988476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5md5v\" (UniqueName: \"kubernetes.io/projected/73a5bfcd-89db-4f2f-8867-dcd39198adf9-kube-api-access-5md5v\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk675f\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.025886 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.025857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:06.089669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.089644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a5bfcd-89db-4f2f-8867-dcd39198adf9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk675f\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.089797 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.089674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md5v\" (UniqueName: \"kubernetes.io/projected/73a5bfcd-89db-4f2f-8867-dcd39198adf9-kube-api-access-5md5v\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk675f\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.092175 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.092146    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a5bfcd-89db-4f2f-8867-dcd39198adf9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk675f\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.097714 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.097683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md5v\" (UniqueName: \"kubernetes.io/projected/73a5bfcd-89db-4f2f-8867-dcd39198adf9-kube-api-access-5md5v\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedk675f\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.240636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.240558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.298213 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.298181    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"]
Sep 19 08:44:06.306661 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:44:06.306601    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59304542_9e02_46b6_ae52_7a07aa7bd940.slice/crio-4b314cf01775150fb25af21c0d348c41ed2539331218ac5f9aa5714ee5061a49 WatchSource:0}: Error finding container 4b314cf01775150fb25af21c0d348c41ed2539331218ac5f9aa5714ee5061a49: Status 404 returned error can't find the container with id 4b314cf01775150fb25af21c0d348c41ed2539331218ac5f9aa5714ee5061a49
Sep 19 08:44:06.524692 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.524670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"]
Sep 19 08:44:06.527239 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:44:06.527197    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a5bfcd_89db_4f2f_8867_dcd39198adf9.slice/crio-e6fa2b4e1802a998b5b9a783981784a34f07f9df979d2c145d2a7715a7bc529c WatchSource:0}: Error finding container e6fa2b4e1802a998b5b9a783981784a34f07f9df979d2c145d2a7715a7bc529c: Status 404 returned error can't find the container with id e6fa2b4e1802a998b5b9a783981784a34f07f9df979d2c145d2a7715a7bc529c
Sep 19 08:44:06.849521 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.849425    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" event={"ID":"73a5bfcd-89db-4f2f-8867-dcd39198adf9","Type":"ContainerStarted","Data":"ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7"}
Sep 19 08:44:06.851045 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.851020    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:06.851250 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.851230    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" event={"ID":"73a5bfcd-89db-4f2f-8867-dcd39198adf9","Type":"ContainerStarted","Data":"e6fa2b4e1802a998b5b9a783981784a34f07f9df979d2c145d2a7715a7bc529c"}
Sep 19 08:44:06.852474 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.852444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" event={"ID":"59304542-9e02-46b6-ae52-7a07aa7bd940","Type":"ContainerStarted","Data":"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"}
Sep 19 08:44:06.852572 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.852513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" event={"ID":"59304542-9e02-46b6-ae52-7a07aa7bd940","Type":"ContainerStarted","Data":"4b314cf01775150fb25af21c0d348c41ed2539331218ac5f9aa5714ee5061a49"}
Sep 19 08:44:06.853976 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.853955    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:06.871603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:06.871554    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" podStartSLOduration=1.871536487 podStartE2EDuration="1.871536487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:06.868677598 +0000 UTC m=+1399.054869555" watchObservedRunningTime="2026-09-19 08:44:06.871536487 +0000 UTC m=+1399.057728444"
Sep 19 08:44:07.858701 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:07.858665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" event={"ID":"59304542-9e02-46b6-ae52-7a07aa7bd940","Type":"ContainerStarted","Data":"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"}
Sep 19 08:44:07.879965 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:07.879913    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" podStartSLOduration=2.8798979940000002 podStartE2EDuration="2.879897994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:07.877961012 +0000 UTC m=+1400.064152980" watchObservedRunningTime="2026-09-19 08:44:07.879897994 +0000 UTC m=+1400.066089952"
Sep 19 08:44:08.026614 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:08.026573    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:16.026300 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:16.026259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:16.039770 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:16.039743    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:16.039963 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:16.039931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:16.912184 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:16.912157    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:25.773952 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.773914    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"]
Sep 19 08:44:25.777808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.777790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:25.781209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.781153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-th9bg\""
Sep 19 08:44:25.781209 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.781166    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 19 08:44:25.781422 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.781178    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 19 08:44:25.781422 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.781246    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-pws2b\""
Sep 19 08:44:25.781422 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.781156    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 19 08:44:25.786906 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.786884    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"]
Sep 19 08:44:25.952301 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.952272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jtv\" (UniqueName: \"kubernetes.io/projected/fe3507a5-2e9e-42e3-ba9a-6c2893431583-kube-api-access-72jtv\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:25.952470 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:25.952320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3507a5-2e9e-42e3-ba9a-6c2893431583-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.052856 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.052772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3507a5-2e9e-42e3-ba9a-6c2893431583-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.052856 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.052849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jtv\" (UniqueName: \"kubernetes.io/projected/fe3507a5-2e9e-42e3-ba9a-6c2893431583-kube-api-access-72jtv\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.055453 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.055424    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3507a5-2e9e-42e3-ba9a-6c2893431583-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.060461 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.060439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jtv\" (UniqueName: \"kubernetes.io/projected/fe3507a5-2e9e-42e3-ba9a-6c2893431583-kube-api-access-72jtv\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.088997 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.088969    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.357399 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.357317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"]
Sep 19 08:44:26.360805 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:44:26.360276    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3507a5_2e9e_42e3_ba9a_6c2893431583.slice/crio-f9a58a89031f36ad2cfe79d65d2576d18ff3fdbb3da513cad412b613f4ff167a WatchSource:0}: Error finding container f9a58a89031f36ad2cfe79d65d2576d18ff3fdbb3da513cad412b613f4ff167a: Status 404 returned error can't find the container with id f9a58a89031f36ad2cfe79d65d2576d18ff3fdbb3da513cad412b613f4ff167a
Sep 19 08:44:26.941028 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.940984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" event={"ID":"fe3507a5-2e9e-42e3-ba9a-6c2893431583","Type":"ContainerStarted","Data":"820eb079f5cc4db2e8677e7217c531e96931e3726cb755ca0065791b7a082948"}
Sep 19 08:44:26.941518 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.941058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" event={"ID":"fe3507a5-2e9e-42e3-ba9a-6c2893431583","Type":"ContainerStarted","Data":"f9a58a89031f36ad2cfe79d65d2576d18ff3fdbb3da513cad412b613f4ff167a"}
Sep 19 08:44:26.941518 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.941094    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:44:26.959896 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:26.959852    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" podStartSLOduration=1.959835076 podStartE2EDuration="1.959835076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:26.957970883 +0000 UTC m=+1419.144162841" watchObservedRunningTime="2026-09-19 08:44:26.959835076 +0000 UTC m=+1419.146027066"
Sep 19 08:44:37.860460 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:37.860385    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:44:52.426989 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.426956    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"]
Sep 19 08:44:52.436640 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:44:52.436599    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:44:52.436640 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:44:52.436640 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:44:52.436640 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7" cmd=["/bin/sleep","15"]
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.436665    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" message="" err=<
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:44:52.436682    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="main" containerID="cri-o://ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.436732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"]
Sep 19 08:44:52.436906 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.436772    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="main" containerID="cri-o://ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7" gracePeriod=60
Sep 19 08:44:52.649194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.649162    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="llm-d-routing-sidecar" containerID="cri-o://3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744" gracePeriod=60
Sep 19 08:44:52.891047 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.891023    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:52.990427 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-home\") pod \"59304542-9e02-46b6-ae52-7a07aa7bd940\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") "
Sep 19 08:44:52.990427 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59304542-9e02-46b6-ae52-7a07aa7bd940-tls-certs\") pod \"59304542-9e02-46b6-ae52-7a07aa7bd940\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") "
Sep 19 08:44:52.990669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-model-cache\") pod \"59304542-9e02-46b6-ae52-7a07aa7bd940\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") "
Sep 19 08:44:52.990669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-tmp-dir\") pod \"59304542-9e02-46b6-ae52-7a07aa7bd940\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") "
Sep 19 08:44:52.990669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990571    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-dshm\") pod \"59304542-9e02-46b6-ae52-7a07aa7bd940\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") "
Sep 19 08:44:52.990669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pkfs\" (UniqueName: \"kubernetes.io/projected/59304542-9e02-46b6-ae52-7a07aa7bd940-kube-api-access-8pkfs\") pod \"59304542-9e02-46b6-ae52-7a07aa7bd940\" (UID: \"59304542-9e02-46b6-ae52-7a07aa7bd940\") "
Sep 19 08:44:52.990878 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990741    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-home" (OuterVolumeSpecName: "home") pod "59304542-9e02-46b6-ae52-7a07aa7bd940" (UID: "59304542-9e02-46b6-ae52-7a07aa7bd940"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:52.990878 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990764    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-model-cache" (OuterVolumeSpecName: "model-cache") pod "59304542-9e02-46b6-ae52-7a07aa7bd940" (UID: "59304542-9e02-46b6-ae52-7a07aa7bd940"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:52.991003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.990909    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59304542-9e02-46b6-ae52-7a07aa7bd940" (UID: "59304542-9e02-46b6-ae52-7a07aa7bd940"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:52.992747 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.992719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-dshm" (OuterVolumeSpecName: "dshm") pod "59304542-9e02-46b6-ae52-7a07aa7bd940" (UID: "59304542-9e02-46b6-ae52-7a07aa7bd940"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:44:52.992851 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.992811    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59304542-9e02-46b6-ae52-7a07aa7bd940-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59304542-9e02-46b6-ae52-7a07aa7bd940" (UID: "59304542-9e02-46b6-ae52-7a07aa7bd940"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:44:52.992897 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:52.992852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59304542-9e02-46b6-ae52-7a07aa7bd940-kube-api-access-8pkfs" (OuterVolumeSpecName: "kube-api-access-8pkfs") pod "59304542-9e02-46b6-ae52-7a07aa7bd940" (UID: "59304542-9e02-46b6-ae52-7a07aa7bd940"). InnerVolumeSpecName "kube-api-access-8pkfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:44:53.057683 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057607    2580 generic.go:356] "Generic (PLEG): container finished" podID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerID="ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7" exitCode=0
Sep 19 08:44:53.057683 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057635    2580 generic.go:356] "Generic (PLEG): container finished" podID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerID="3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744" exitCode=0
Sep 19 08:44:53.057683 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057672    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"
Sep 19 08:44:53.057869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" event={"ID":"59304542-9e02-46b6-ae52-7a07aa7bd940","Type":"ContainerDied","Data":"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"}
Sep 19 08:44:53.057869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057754    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" event={"ID":"59304542-9e02-46b6-ae52-7a07aa7bd940","Type":"ContainerDied","Data":"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"}
Sep 19 08:44:53.057869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn" event={"ID":"59304542-9e02-46b6-ae52-7a07aa7bd940","Type":"ContainerDied","Data":"4b314cf01775150fb25af21c0d348c41ed2539331218ac5f9aa5714ee5061a49"}
Sep 19 08:44:53.057869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.057795    2580 scope.go:117] "RemoveContainer" containerID="ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"
Sep 19 08:44:53.066539 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.066523    2580 scope.go:117] "RemoveContainer" containerID="3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"
Sep 19 08:44:53.079468 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.079431    2580 scope.go:117] "RemoveContainer" containerID="ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"
Sep 19 08:44:53.079755 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:44:53.079733    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7\": container with ID starting with ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7 not found: ID does not exist" containerID="ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"
Sep 19 08:44:53.079815 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.079764    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"} err="failed to get container status \"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7\": rpc error: code = NotFound desc = could not find container \"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7\": container with ID starting with ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7 not found: ID does not exist"
Sep 19 08:44:53.079815 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.079780    2580 scope.go:117] "RemoveContainer" containerID="3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"
Sep 19 08:44:53.080033 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:44:53.080016    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744\": container with ID starting with 3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744 not found: ID does not exist" containerID="3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"
Sep 19 08:44:53.080070 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.080038    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"} err="failed to get container status \"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744\": rpc error: code = NotFound desc = could not find container \"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744\": container with ID starting with 3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744 not found: ID does not exist"
Sep 19 08:44:53.080070 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.080052    2580 scope.go:117] "RemoveContainer" containerID="ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"
Sep 19 08:44:53.080135 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.080103    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"]
Sep 19 08:44:53.080322 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.080304    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7"} err="failed to get container status \"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7\": rpc error: code = NotFound desc = could not find container \"ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7\": container with ID starting with ae6655d1ca87d131bd7214bdf768ce5641f438ca677eccdef21e12e47978c9e7 not found: ID does not exist"
Sep 19 08:44:53.080372 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.080323    2580 scope.go:117] "RemoveContainer" containerID="3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"
Sep 19 08:44:53.080558 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.080540    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744"} err="failed to get container status \"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744\": rpc error: code = NotFound desc = could not find container \"3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744\": container with ID starting with 3fcccb307b10039a062949c78af4c4d018b37cc55be1ace4fdfdb61ca2c22744 not found: ID does not exist"
Sep 19 08:44:53.084269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.084211    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-856bdf4d56-ndsxn"]
Sep 19 08:44:53.091601 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.091582    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkfs\" (UniqueName: \"kubernetes.io/projected/59304542-9e02-46b6-ae52-7a07aa7bd940-kube-api-access-8pkfs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:44:53.091668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.091603    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:44:53.091668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.091613    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59304542-9e02-46b6-ae52-7a07aa7bd940-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:44:53.091668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.091622    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:44:53.091668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.091646    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:44:53.091668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:53.091653    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59304542-9e02-46b6-ae52-7a07aa7bd940-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:44:54.079179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:54.079145    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" path="/var/lib/kubelet/pods/59304542-9e02-46b6-ae52-7a07aa7bd940/volumes"
Sep 19 08:44:57.861324 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:57.861277    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:44:57.947309 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:44:57.947283    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:45:07.455438 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:07.455393    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main" containerID="cri-o://ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7" gracePeriod=45
Sep 19 08:45:07.860842 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:07.860755    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:45:17.860936 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:17.860892    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:45:17.861355 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:17.860999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:45:27.860554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:27.860511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:45:37.693737 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.693710    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:45:37.761577 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.761506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a5bfcd-89db-4f2f-8867-dcd39198adf9-tls-certs\") pod \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") "
Sep 19 08:45:37.761577 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.761560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5md5v\" (UniqueName: \"kubernetes.io/projected/73a5bfcd-89db-4f2f-8867-dcd39198adf9-kube-api-access-5md5v\") pod \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\" (UID: \"73a5bfcd-89db-4f2f-8867-dcd39198adf9\") "
Sep 19 08:45:37.763622 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.763597    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a5bfcd-89db-4f2f-8867-dcd39198adf9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "73a5bfcd-89db-4f2f-8867-dcd39198adf9" (UID: "73a5bfcd-89db-4f2f-8867-dcd39198adf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:45:37.763748 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.763637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a5bfcd-89db-4f2f-8867-dcd39198adf9-kube-api-access-5md5v" (OuterVolumeSpecName: "kube-api-access-5md5v") pod "73a5bfcd-89db-4f2f-8867-dcd39198adf9" (UID: "73a5bfcd-89db-4f2f-8867-dcd39198adf9"). InnerVolumeSpecName "kube-api-access-5md5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:45:37.862255 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.862208    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73a5bfcd-89db-4f2f-8867-dcd39198adf9-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:45:37.862255 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:37.862253    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5md5v\" (UniqueName: \"kubernetes.io/projected/73a5bfcd-89db-4f2f-8867-dcd39198adf9-kube-api-access-5md5v\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:45:38.250266 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.250211    2580 generic.go:356] "Generic (PLEG): container finished" podID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerID="ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7" exitCode=0
Sep 19 08:45:38.250403 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.250272    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"
Sep 19 08:45:38.250403 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.250287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" event={"ID":"73a5bfcd-89db-4f2f-8867-dcd39198adf9","Type":"ContainerDied","Data":"ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7"}
Sep 19 08:45:38.250403 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.250331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f" event={"ID":"73a5bfcd-89db-4f2f-8867-dcd39198adf9","Type":"ContainerDied","Data":"e6fa2b4e1802a998b5b9a783981784a34f07f9df979d2c145d2a7715a7bc529c"}
Sep 19 08:45:38.250403 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.250351    2580 scope.go:117] "RemoveContainer" containerID="ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7"
Sep 19 08:45:38.258438 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.258421    2580 scope.go:117] "RemoveContainer" containerID="ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7"
Sep 19 08:45:38.258696 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:45:38.258679    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7\": container with ID starting with ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7 not found: ID does not exist" containerID="ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7"
Sep 19 08:45:38.258788 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.258704    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7"} err="failed to get container status \"ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7\": rpc error: code = NotFound desc = could not find container \"ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7\": container with ID starting with ad43a2fcb73957412b8e6981603abee747ff11edf66083f114409f6bdf4b1bf7 not found: ID does not exist"
Sep 19 08:45:38.266997 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.266976    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"]
Sep 19 08:45:38.271370 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:38.271348    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedk675f"]
Sep 19 08:45:40.079139 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:40.079111    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" path="/var/lib/kubelet/pods/73a5bfcd-89db-4f2f-8867-dcd39198adf9/volumes"
Sep 19 08:45:44.797139 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797108    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"]
Sep 19 08:45:44.797628 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797588    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main"
Sep 19 08:45:44.797628 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797605    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main"
Sep 19 08:45:44.797745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797632    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="llm-d-routing-sidecar"
Sep 19 08:45:44.797745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797641    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="llm-d-routing-sidecar"
Sep 19 08:45:44.797745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797655    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="main"
Sep 19 08:45:44.797745 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="main"
Sep 19 08:45:44.797949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797745    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="main"
Sep 19 08:45:44.797949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797757    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59304542-9e02-46b6-ae52-7a07aa7bd940" containerName="llm-d-routing-sidecar"
Sep 19 08:45:44.797949 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.797769    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a5bfcd-89db-4f2f-8867-dcd39198adf9" containerName="main"
Sep 19 08:45:44.801023 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.801001    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.803381 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.803355    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 19 08:45:44.803479 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.803391    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 19 08:45:44.803479 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.803401    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 19 08:45:44.804596 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.804574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 19 08:45:44.804688 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.804583    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-z776q\""
Sep 19 08:45:44.804688 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.804585    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:45:44.812680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.812660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"]
Sep 19 08:45:44.920196 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920162    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7474s\" (UniqueName: \"kubernetes.io/projected/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kube-api-access-7474s\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920376    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:44.920611 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:44.920412    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.021624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.021817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.021817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7474s\" (UniqueName: \"kubernetes.io/projected/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kube-api-access-7474s\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.021948 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.021948 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.021948 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022108 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.021971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022108 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.022015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022215 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.022199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022305 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.022256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022425 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.022316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022515 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.022494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.022722 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.022690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.024145 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.024125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.024567 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.024550    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.036578 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.036555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7474s\" (UniqueName: \"kubernetes.io/projected/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kube-api-access-7474s\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-fz6pd\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.049198 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.049136    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"]
Sep 19 08:45:45.053553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.053530    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.055826 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.055808    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-7q8p2\""
Sep 19 08:45:45.064038 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.064012    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"]
Sep 19 08:45:45.110766 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.110737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:45.122749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.122725    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7454e114-2020-42e3-a349-628459ff3809-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.122843 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.122783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwngp\" (UniqueName: \"kubernetes.io/projected/7454e114-2020-42e3-a349-628459ff3809-kube-api-access-mwngp\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.223372 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.223340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7454e114-2020-42e3-a349-628459ff3809-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.223540 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.223399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwngp\" (UniqueName: \"kubernetes.io/projected/7454e114-2020-42e3-a349-628459ff3809-kube-api-access-mwngp\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.225923 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.225892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7454e114-2020-42e3-a349-628459ff3809-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.231125 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.231104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwngp\" (UniqueName: \"kubernetes.io/projected/7454e114-2020-42e3-a349-628459ff3809-kube-api-access-mwngp\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.364147 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.364069    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:45.383087 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.383056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"]
Sep 19 08:45:45.386542 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:45:45.386519    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f172888_fcdb_4d63_bc07_4c6a28dd7bbf.slice/crio-830fa6100cd860c5c8c0b551e438c9c405fcb5ead51a1a38526550c7128133c5 WatchSource:0}: Error finding container 830fa6100cd860c5c8c0b551e438c9c405fcb5ead51a1a38526550c7128133c5: Status 404 returned error can't find the container with id 830fa6100cd860c5c8c0b551e438c9c405fcb5ead51a1a38526550c7128133c5
Sep 19 08:45:45.635189 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:45.635112    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"]
Sep 19 08:45:45.638091 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:45:45.638064    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7454e114_2020_42e3_a349_628459ff3809.slice/crio-aaf7ad60c71f65239fd9df628ab418d82e8841af65f865a9b9e5be33b4b0bbc5 WatchSource:0}: Error finding container aaf7ad60c71f65239fd9df628ab418d82e8841af65f865a9b9e5be33b4b0bbc5: Status 404 returned error can't find the container with id aaf7ad60c71f65239fd9df628ab418d82e8841af65f865a9b9e5be33b4b0bbc5
Sep 19 08:45:46.289887 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:46.289845    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" event={"ID":"7454e114-2020-42e3-a349-628459ff3809","Type":"ContainerStarted","Data":"a3897512d17da39675403141985eeb89738aa3a7390d9a255fb2c9a8949711f3"}
Sep 19 08:45:46.290398 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:46.289915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" event={"ID":"7454e114-2020-42e3-a349-628459ff3809","Type":"ContainerStarted","Data":"aaf7ad60c71f65239fd9df628ab418d82e8841af65f865a9b9e5be33b4b0bbc5"}
Sep 19 08:45:46.290398 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:46.290100    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:45:46.292105 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:46.292080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" event={"ID":"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf","Type":"ContainerStarted","Data":"830fa6100cd860c5c8c0b551e438c9c405fcb5ead51a1a38526550c7128133c5"}
Sep 19 08:45:46.309097 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:46.309046    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podStartSLOduration=1.309027926 podStartE2EDuration="1.309027926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:45:46.306333879 +0000 UTC m=+1498.492525837" watchObservedRunningTime="2026-09-19 08:45:46.309027926 +0000 UTC m=+1498.495219880"
Sep 19 08:45:49.307715 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:49.307620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" event={"ID":"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf","Type":"ContainerStarted","Data":"4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca"}
Sep 19 08:45:51.318344 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:51.318306    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerID="4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca" exitCode=0
Sep 19 08:45:51.318715 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:51.318380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" event={"ID":"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf","Type":"ContainerDied","Data":"4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca"}
Sep 19 08:45:52.324624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:52.324586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" event={"ID":"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf","Type":"ContainerStarted","Data":"3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"}
Sep 19 08:45:52.344966 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:52.344904    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" podStartSLOduration=2.632824866 podStartE2EDuration="8.344889013s" totalImagesPullingTime="3.61729362s" totalInitContainerRuntime="2.094770527s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:45:52.343555317 +0000 UTC m=+1504.529747287" watchObservedRunningTime="2026-09-19 08:45:52.344889013 +0000 UTC m=+1504.531081022"
Sep 19 08:45:53.111809 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:53.111757    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:55.111206 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:55.111166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:55.123668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:55.123643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:45:55.349499 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:45:55.349469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:46:17.298771 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:17.298690    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:46:22.414837 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.414397    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"]
Sep 19 08:46:22.417773 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.417744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"]
Sep 19 08:46:22.431457 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:46:22.431418    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:46:22.431457 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:46:22.431457 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:46:22.431457 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9" cmd=["/bin/sleep","15"]
Sep 19 08:46:22.431838 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.431821    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" message="" err=<
Sep 19 08:46:22.431838 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:46:22.431838 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:46:22.431838 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:46:22.432070 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:46:22.432055    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:46:22.432070 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:46:22.432070 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:46:22.432070 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="main" containerID="cri-o://3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"
Sep 19 08:46:22.432368 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.432350    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="main" containerID="cri-o://3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9" gracePeriod=60
Sep 19 08:46:22.786046 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.786020    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:46:22.867524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kserve-provision-location\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tls-certs\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-dshm\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-model-cache\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-home\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7474s\" (UniqueName: \"kubernetes.io/projected/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kube-api-access-7474s\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tmp-dir\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.867792 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867776    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-cabundle-cert\") pod \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\" (UID: \"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf\") "
Sep 19 08:46:22.868115 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.867958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-home" (OuterVolumeSpecName: "home") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.868115 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.868081    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.868242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.868156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.868347 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.868323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:46:22.869940 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.869914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-dshm" (OuterVolumeSpecName: "dshm") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.869940 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.869931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kube-api-access-7474s" (OuterVolumeSpecName: "kube-api-access-7474s") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "kube-api-access-7474s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:46:22.870126 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.870003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:46:22.887396 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.887363    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" (UID: "6f172888-fcdb-4d63-bc07-4c6a28dd7bbf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:46:22.968418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968343    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968373    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7474s\" (UniqueName: \"kubernetes.io/projected/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kube-api-access-7474s\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968402    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968412    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968420    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968644 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968427    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968644 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968435    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:22.968644 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:22.968443    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:46:23.464487 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.464454    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerID="3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9" exitCode=0
Sep 19 08:46:23.464914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.464500    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"
Sep 19 08:46:23.464914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.464522    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" event={"ID":"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf","Type":"ContainerDied","Data":"3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"}
Sep 19 08:46:23.464914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.464568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd" event={"ID":"6f172888-fcdb-4d63-bc07-4c6a28dd7bbf","Type":"ContainerDied","Data":"830fa6100cd860c5c8c0b551e438c9c405fcb5ead51a1a38526550c7128133c5"}
Sep 19 08:46:23.464914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.464591    2580 scope.go:117] "RemoveContainer" containerID="3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"
Sep 19 08:46:23.474393 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.474375    2580 scope.go:117] "RemoveContainer" containerID="4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca"
Sep 19 08:46:23.481796 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.481778    2580 scope.go:117] "RemoveContainer" containerID="3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"
Sep 19 08:46:23.482058 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:46:23.482038    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9\": container with ID starting with 3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9 not found: ID does not exist" containerID="3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"
Sep 19 08:46:23.482114 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.482066    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9"} err="failed to get container status \"3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9\": rpc error: code = NotFound desc = could not find container \"3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9\": container with ID starting with 3b4257e1bf72aee3df1f33499efa14fae1789f4c84fbd608282c64f889565fd9 not found: ID does not exist"
Sep 19 08:46:23.482114 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.482092    2580 scope.go:117] "RemoveContainer" containerID="4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca"
Sep 19 08:46:23.482354 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:46:23.482338    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca\": container with ID starting with 4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca not found: ID does not exist" containerID="4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca"
Sep 19 08:46:23.482410 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.482359    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca"} err="failed to get container status \"4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca\": rpc error: code = NotFound desc = could not find container \"4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca\": container with ID starting with 4c32fb4ae3b60cf8d6c69fee0eaf3fbddf7e909324a61aef85f267e6120346ca not found: ID does not exist"
Sep 19 08:46:23.486242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.486207    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"]
Sep 19 08:46:23.490305 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:23.490283    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-fz6pd"]
Sep 19 08:46:24.080121 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:24.080073    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" path="/var/lib/kubelet/pods/6f172888-fcdb-4d63-bc07-4c6a28dd7bbf/volumes"
Sep 19 08:46:27.299164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:27.299127    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:46:37.298645 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:37.298599    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:46:37.438882 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:37.438833    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main" containerID="cri-o://a3897512d17da39675403141985eeb89738aa3a7390d9a255fb2c9a8949711f3" gracePeriod=45
Sep 19 08:46:47.299000 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:47.298956    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:46:47.299606 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:47.299070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:46:57.299491 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:46:57.299453    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:47:00.680214 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:00.680181    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"]
Sep 19 08:47:07.299333 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.299293    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:47:07.664298 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.664269    2580 generic.go:356] "Generic (PLEG): container finished" podID="7454e114-2020-42e3-a349-628459ff3809" containerID="a3897512d17da39675403141985eeb89738aa3a7390d9a255fb2c9a8949711f3" exitCode=0
Sep 19 08:47:07.664447 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.664326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" event={"ID":"7454e114-2020-42e3-a349-628459ff3809","Type":"ContainerDied","Data":"a3897512d17da39675403141985eeb89738aa3a7390d9a255fb2c9a8949711f3"}
Sep 19 08:47:07.680273 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.680251    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:47:07.754545 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.754516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7454e114-2020-42e3-a349-628459ff3809-tls-certs\") pod \"7454e114-2020-42e3-a349-628459ff3809\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") "
Sep 19 08:47:07.754712 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.754564    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwngp\" (UniqueName: \"kubernetes.io/projected/7454e114-2020-42e3-a349-628459ff3809-kube-api-access-mwngp\") pod \"7454e114-2020-42e3-a349-628459ff3809\" (UID: \"7454e114-2020-42e3-a349-628459ff3809\") "
Sep 19 08:47:07.756647 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.756616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7454e114-2020-42e3-a349-628459ff3809-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7454e114-2020-42e3-a349-628459ff3809" (UID: "7454e114-2020-42e3-a349-628459ff3809"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:47:07.756762 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.756650    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7454e114-2020-42e3-a349-628459ff3809-kube-api-access-mwngp" (OuterVolumeSpecName: "kube-api-access-mwngp") pod "7454e114-2020-42e3-a349-628459ff3809" (UID: "7454e114-2020-42e3-a349-628459ff3809"). InnerVolumeSpecName "kube-api-access-mwngp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:47:07.855273 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.855170    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7454e114-2020-42e3-a349-628459ff3809-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:47:07.855273 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.855201    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwngp\" (UniqueName: \"kubernetes.io/projected/7454e114-2020-42e3-a349-628459ff3809-kube-api-access-mwngp\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:47:07.946969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:07.946927    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:47:08.669839 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:08.669807    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"
Sep 19 08:47:08.669839 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:08.669816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629" event={"ID":"7454e114-2020-42e3-a349-628459ff3809","Type":"ContainerDied","Data":"aaf7ad60c71f65239fd9df628ab418d82e8841af65f865a9b9e5be33b4b0bbc5"}
Sep 19 08:47:08.670402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:08.669870    2580 scope.go:117] "RemoveContainer" containerID="a3897512d17da39675403141985eeb89738aa3a7390d9a255fb2c9a8949711f3"
Sep 19 08:47:08.686080 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:08.686057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"]
Sep 19 08:47:08.688270 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:08.688251    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69d4629"]
Sep 19 08:47:10.080233 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:10.080195    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7454e114-2020-42e3-a349-628459ff3809" path="/var/lib/kubelet/pods/7454e114-2020-42e3-a349-628459ff3809/volumes"
Sep 19 08:47:15.700050 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:15.699986    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main" containerID="cri-o://820eb079f5cc4db2e8677e7217c531e96931e3726cb755ca0065791b7a082948" gracePeriod=45
Sep 19 08:47:17.104899 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.104866    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"]
Sep 19 08:47:17.105306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105198    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="main"
Sep 19 08:47:17.105306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105211    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="main"
Sep 19 08:47:17.105306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105234    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main"
Sep 19 08:47:17.105306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105243    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main"
Sep 19 08:47:17.105306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105257    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="storage-initializer"
Sep 19 08:47:17.105306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105263    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="storage-initializer"
Sep 19 08:47:17.105502 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105326    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7454e114-2020-42e3-a349-628459ff3809" containerName="main"
Sep 19 08:47:17.105502 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.105334    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f172888-fcdb-4d63-bc07-4c6a28dd7bbf" containerName="main"
Sep 19 08:47:17.108489 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.108471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.111264 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.111215    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 19 08:47:17.111382 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.111266    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 19 08:47:17.112262 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.112242    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:47:17.112357 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.112311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 19 08:47:17.112415 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.112399    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-769mf\""
Sep 19 08:47:17.112472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.112457    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 19 08:47:17.123658 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.123623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"]
Sep 19 08:47:17.242197 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242401 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242203    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242401 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242276    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242401 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242299    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242401 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242615 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f83faef-82c8-46b5-8a35-2c372f87c3f1-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242615 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.242615 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.242509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pkdg\" (UniqueName: \"kubernetes.io/projected/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kube-api-access-5pkdg\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343541 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f83faef-82c8-46b5-8a35-2c372f87c3f1-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.343779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pkdg\" (UniqueName: \"kubernetes.io/projected/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kube-api-access-5pkdg\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.344164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.343982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.344164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.344131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.344321 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.344174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.344321 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.344215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.344449 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.344430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f83faef-82c8-46b5-8a35-2c372f87c3f1-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.346103 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.346045    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.346289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.346272    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.352859 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.352835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pkdg\" (UniqueName: \"kubernetes.io/projected/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kube-api-access-5pkdg\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.418876 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.418812    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:17.547749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.547720    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"]
Sep 19 08:47:17.551347 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.551325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.553692 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.553668    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-7rsnn\""
Sep 19 08:47:17.559772 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.559751    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"]
Sep 19 08:47:17.646177 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.646144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.646343 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.646194    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svbwf\" (UniqueName: \"kubernetes.io/projected/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-kube-api-access-svbwf\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.698859 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.698829    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"]
Sep 19 08:47:17.702860 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:47:17.702823    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f83faef_82c8_46b5_8a35_2c372f87c3f1.slice/crio-d77b03cbd68d5da3c01d1ca3a1f29c2ea47da77192604ecef199ebe78deaeada WatchSource:0}: Error finding container d77b03cbd68d5da3c01d1ca3a1f29c2ea47da77192604ecef199ebe78deaeada: Status 404 returned error can't find the container with id d77b03cbd68d5da3c01d1ca3a1f29c2ea47da77192604ecef199ebe78deaeada
Sep 19 08:47:17.708580 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.708536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" event={"ID":"9f83faef-82c8-46b5-8a35-2c372f87c3f1","Type":"ContainerStarted","Data":"d77b03cbd68d5da3c01d1ca3a1f29c2ea47da77192604ecef199ebe78deaeada"}
Sep 19 08:47:17.747306 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.747283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.747439 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.747352    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svbwf\" (UniqueName: \"kubernetes.io/projected/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-kube-api-access-svbwf\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.749678 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.749657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.755381 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.755352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svbwf\" (UniqueName: \"kubernetes.io/projected/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-kube-api-access-svbwf\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.862056 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.862022    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:17.947876 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:17.947840    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:47:18.132704 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:18.132668    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"]
Sep 19 08:47:18.135013 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:47:18.134982    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56cc31c_52a0_4f2c_b2da_66d1dba58f02.slice/crio-a1d2624cb2f6fbeb9713c49c56cec1ec97fe08ed5446d5368a9ea08c8d324a9f WatchSource:0}: Error finding container a1d2624cb2f6fbeb9713c49c56cec1ec97fe08ed5446d5368a9ea08c8d324a9f: Status 404 returned error can't find the container with id a1d2624cb2f6fbeb9713c49c56cec1ec97fe08ed5446d5368a9ea08c8d324a9f
Sep 19 08:47:18.714124 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:18.714087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" event={"ID":"c56cc31c-52a0-4f2c-b2da-66d1dba58f02","Type":"ContainerStarted","Data":"74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087"}
Sep 19 08:47:18.714350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:18.714140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" event={"ID":"c56cc31c-52a0-4f2c-b2da-66d1dba58f02","Type":"ContainerStarted","Data":"a1d2624cb2f6fbeb9713c49c56cec1ec97fe08ed5446d5368a9ea08c8d324a9f"}
Sep 19 08:47:18.714350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:18.714190    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:47:18.715744 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:18.715723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" event={"ID":"9f83faef-82c8-46b5-8a35-2c372f87c3f1","Type":"ContainerStarted","Data":"a2773bf8ee13ecfeac1c45a352ba6ca8ed237d2056fc352846673f0aa8d581c9"}
Sep 19 08:47:18.729907 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:18.729864    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podStartSLOduration=1.72984932 podStartE2EDuration="1.72984932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:47:18.728305887 +0000 UTC m=+1590.914497845" watchObservedRunningTime="2026-09-19 08:47:18.72984932 +0000 UTC m=+1590.916041304"
Sep 19 08:47:20.726701 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:20.726663    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerID="a2773bf8ee13ecfeac1c45a352ba6ca8ed237d2056fc352846673f0aa8d581c9" exitCode=0
Sep 19 08:47:20.727106 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:20.726712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" event={"ID":"9f83faef-82c8-46b5-8a35-2c372f87c3f1","Type":"ContainerDied","Data":"a2773bf8ee13ecfeac1c45a352ba6ca8ed237d2056fc352846673f0aa8d581c9"}
Sep 19 08:47:21.733140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:21.733103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" event={"ID":"9f83faef-82c8-46b5-8a35-2c372f87c3f1","Type":"ContainerStarted","Data":"6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877"}
Sep 19 08:47:21.753870 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:21.753819    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podStartSLOduration=2.52133259 podStartE2EDuration="4.7538041s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.23247151s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:47:21.751884396 +0000 UTC m=+1593.938076352" watchObservedRunningTime="2026-09-19 08:47:21.7538041 +0000 UTC m=+1593.939996057"
Sep 19 08:47:22.418915 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:22.418875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:27.419602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:27.419563    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:27.432175 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:27.432151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:27.769754 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:27.769727    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:47:27.947866 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:27.947829    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:47:27.948030 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:27.947925    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:47:37.946853 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:37.946775    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:47:45.844467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.844422    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerID="820eb079f5cc4db2e8677e7217c531e96931e3726cb755ca0065791b7a082948" exitCode=0
Sep 19 08:47:45.844467 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.844457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" event={"ID":"fe3507a5-2e9e-42e3-ba9a-6c2893431583","Type":"ContainerDied","Data":"820eb079f5cc4db2e8677e7217c531e96931e3726cb755ca0065791b7a082948"}
Sep 19 08:47:45.955248 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.955210    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:47:45.974510 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.974445    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3507a5-2e9e-42e3-ba9a-6c2893431583-tls-certs\") pod \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") "
Sep 19 08:47:45.974510 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.974490    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72jtv\" (UniqueName: \"kubernetes.io/projected/fe3507a5-2e9e-42e3-ba9a-6c2893431583-kube-api-access-72jtv\") pod \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\" (UID: \"fe3507a5-2e9e-42e3-ba9a-6c2893431583\") "
Sep 19 08:47:45.976722 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.976689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3507a5-2e9e-42e3-ba9a-6c2893431583-kube-api-access-72jtv" (OuterVolumeSpecName: "kube-api-access-72jtv") pod "fe3507a5-2e9e-42e3-ba9a-6c2893431583" (UID: "fe3507a5-2e9e-42e3-ba9a-6c2893431583"). InnerVolumeSpecName "kube-api-access-72jtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:47:45.976722 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:45.976709    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3507a5-2e9e-42e3-ba9a-6c2893431583-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe3507a5-2e9e-42e3-ba9a-6c2893431583" (UID: "fe3507a5-2e9e-42e3-ba9a-6c2893431583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:47:46.075154 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.075120    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3507a5-2e9e-42e3-ba9a-6c2893431583-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:47:46.075154 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.075147    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72jtv\" (UniqueName: \"kubernetes.io/projected/fe3507a5-2e9e-42e3-ba9a-6c2893431583-kube-api-access-72jtv\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:47:46.849933 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.849892    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2" event={"ID":"fe3507a5-2e9e-42e3-ba9a-6c2893431583","Type":"ContainerDied","Data":"f9a58a89031f36ad2cfe79d65d2576d18ff3fdbb3da513cad412b613f4ff167a"}
Sep 19 08:47:46.850414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.849951    2580 scope.go:117] "RemoveContainer" containerID="820eb079f5cc4db2e8677e7217c531e96931e3726cb755ca0065791b7a082948"
Sep 19 08:47:46.850414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.849907    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"
Sep 19 08:47:46.866091 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.866068    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"]
Sep 19 08:47:46.869539 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:46.869515    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-4wfg2"]
Sep 19 08:47:48.080440 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:48.080405    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" path="/var/lib/kubelet/pods/fe3507a5-2e9e-42e3-ba9a-6c2893431583/volumes"
Sep 19 08:47:49.723114 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:47:49.723083    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:48:05.995136 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:05.995103    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"]
Sep 19 08:48:05.995617 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:05.995601    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main"
Sep 19 08:48:05.995664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:05.995620    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main"
Sep 19 08:48:05.995726 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:05.995716    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe3507a5-2e9e-42e3-ba9a-6c2893431583" containerName="main"
Sep 19 08:48:06.000476 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.000458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.003015 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.002995    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 19 08:48:06.003271 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.003255    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 19 08:48:06.003733 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.003466    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:48:06.003733 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.003681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 19 08:48:06.003918 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.003784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-6szjw\""
Sep 19 08:48:06.003918 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.003820    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 19 08:48:06.011088 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.011065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"]
Sep 19 08:48:06.045263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045389 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045285    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-dshm\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045389 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045317    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-model-cache\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045389 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxs7p\" (UniqueName: \"kubernetes.io/projected/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kube-api-access-xxs7p\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045498 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tmp-dir\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045498 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045477    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-home\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.045601 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.045581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tls-certs\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146504 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxs7p\" (UniqueName: \"kubernetes.io/projected/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kube-api-access-xxs7p\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146659 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tmp-dir\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146659 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-home\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146659 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146659 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tls-certs\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146659 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146628    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146659 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-dshm\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-model-cache\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146913    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tmp-dir\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.146982 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.146960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-home\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.147149 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.147023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.147206 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.147180    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-model-cache\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.147320 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.147301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.149030 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.149000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-dshm\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.149289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.149275    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tls-certs\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.154542 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.154525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxs7p\" (UniqueName: \"kubernetes.io/projected/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kube-api-access-xxs7p\") pod \"auth-invalid-token-test-kserve-8769cf786-rttgr\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.209336 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.209306    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"]
Sep 19 08:48:06.213664 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.213641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.216070 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.216050    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-cdv7h\""
Sep 19 08:48:06.220703 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.220680    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"]
Sep 19 08:48:06.312354 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.312278    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:06.348529 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.348496    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pr76\" (UniqueName: \"kubernetes.io/projected/f89b6ed9-9998-4015-b719-c082cb132e06-kube-api-access-8pr76\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.348680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.348549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89b6ed9-9998-4015-b719-c082cb132e06-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.449536 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.449502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pr76\" (UniqueName: \"kubernetes.io/projected/f89b6ed9-9998-4015-b719-c082cb132e06-kube-api-access-8pr76\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.449748 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.449553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89b6ed9-9998-4015-b719-c082cb132e06-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.452315 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.452285    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89b6ed9-9998-4015-b719-c082cb132e06-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.459006 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.458984    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pr76\" (UniqueName: \"kubernetes.io/projected/f89b6ed9-9998-4015-b719-c082cb132e06-kube-api-access-8pr76\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.524874 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.524849    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.594683 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.594655    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"]
Sep 19 08:48:06.601263 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:48:06.601202    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd1dbb1_ee12_415b_aeda_5032c1f35cbe.slice/crio-86b6bd87627c30e406a9589f9bf3cd11c08de1a9ab9334bea867b37060a7b3bd WatchSource:0}: Error finding container 86b6bd87627c30e406a9589f9bf3cd11c08de1a9ab9334bea867b37060a7b3bd: Status 404 returned error can't find the container with id 86b6bd87627c30e406a9589f9bf3cd11c08de1a9ab9334bea867b37060a7b3bd
Sep 19 08:48:06.798815 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.798781    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"]
Sep 19 08:48:06.801982 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:48:06.801953    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b6ed9_9998_4015_b719_c082cb132e06.slice/crio-56ba7e433ce1e33d783a03b6e752f8ed2043350e4b3f7003403fa817a4947aa8 WatchSource:0}: Error finding container 56ba7e433ce1e33d783a03b6e752f8ed2043350e4b3f7003403fa817a4947aa8: Status 404 returned error can't find the container with id 56ba7e433ce1e33d783a03b6e752f8ed2043350e4b3f7003403fa817a4947aa8
Sep 19 08:48:06.949954 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.949914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" event={"ID":"f89b6ed9-9998-4015-b719-c082cb132e06","Type":"ContainerStarted","Data":"4f4c3d6a230a2ac751f06ac6ed542e972c81bc9f067759eb6dcd783d85ecd2c3"}
Sep 19 08:48:06.950271 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.950251    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:06.950444 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.950419    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" event={"ID":"f89b6ed9-9998-4015-b719-c082cb132e06","Type":"ContainerStarted","Data":"56ba7e433ce1e33d783a03b6e752f8ed2043350e4b3f7003403fa817a4947aa8"}
Sep 19 08:48:06.952120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.952086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" event={"ID":"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe","Type":"ContainerStarted","Data":"c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c"}
Sep 19 08:48:06.952262 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.952143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" event={"ID":"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe","Type":"ContainerStarted","Data":"86b6bd87627c30e406a9589f9bf3cd11c08de1a9ab9334bea867b37060a7b3bd"}
Sep 19 08:48:06.968096 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:06.968039    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podStartSLOduration=0.968020492 podStartE2EDuration="968.020492ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:48:06.9667045 +0000 UTC m=+1639.152896468" watchObservedRunningTime="2026-09-19 08:48:06.968020492 +0000 UTC m=+1639.154212452"
Sep 19 08:48:08.963791 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:08.963757    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerID="c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c" exitCode=0
Sep 19 08:48:08.964158 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:08.963829    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" event={"ID":"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe","Type":"ContainerDied","Data":"c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c"}
Sep 19 08:48:08.965257 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:08.965204    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:48:15.598363 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:15.598324    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"]
Sep 19 08:48:15.602311 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:15.602278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"]
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:48:15.632115    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877" cmd=["/bin/sleep","15"]
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:15.632184    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" message="" err=<
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:48:15.632200    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" containerID="cri-o://6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877"
Sep 19 08:48:15.633285 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:15.632266    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" containerID="cri-o://6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877" gracePeriod=60
Sep 19 08:48:15.759093 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:15.759035    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 19 08:48:16.760105 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:16.760049    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 19 08:48:16.760611 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:16.760200    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:48:17.759945 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:17.759896    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 19 08:48:18.759595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:18.759476    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 19 08:48:19.723274 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:19.723203    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:48:19.759633 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:19.759593    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 19 08:48:20.040578 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:20.040472    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerID="6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877" exitCode=0
Sep 19 08:48:20.040749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:20.040584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" event={"ID":"9f83faef-82c8-46b5-8a35-2c372f87c3f1","Type":"ContainerDied","Data":"6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877"}
Sep 19 08:48:20.759582 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:20.759431    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 19 08:48:21.038846 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.038824    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:48:21.048310 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.048266    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs" event={"ID":"9f83faef-82c8-46b5-8a35-2c372f87c3f1","Type":"ContainerDied","Data":"d77b03cbd68d5da3c01d1ca3a1f29c2ea47da77192604ecef199ebe78deaeada"}
Sep 19 08:48:21.048443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.048348    2580 scope.go:117] "RemoveContainer" containerID="6e678320b282aa6fe22e2a3495765987a65bafe85a295973936c549fb6f55877"
Sep 19 08:48:21.048443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.048281    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"
Sep 19 08:48:21.059334 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.059046    2580 scope.go:117] "RemoveContainer" containerID="a2773bf8ee13ecfeac1c45a352ba6ca8ed237d2056fc352846673f0aa8d581c9"
Sep 19 08:48:21.198052 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pkdg\" (UniqueName: \"kubernetes.io/projected/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kube-api-access-5pkdg\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198246 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-home\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198246 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198162    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-dshm\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198246 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198209    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f83faef-82c8-46b5-8a35-2c372f87c3f1-cabundle-cert\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-model-cache\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198327    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tmp-dir\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tls-certs\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198394    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kserve-provision-location\") pod \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\" (UID: \"9f83faef-82c8-46b5-8a35-2c372f87c3f1\") "
Sep 19 08:48:21.198668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-home" (OuterVolumeSpecName: "home") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:21.198668 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:21.198778 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198718    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.198778 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198734    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.198880 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.198780    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:21.199527 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.199500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f83faef-82c8-46b5-8a35-2c372f87c3f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:48:21.201004 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.200971    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kube-api-access-5pkdg" (OuterVolumeSpecName: "kube-api-access-5pkdg") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "kube-api-access-5pkdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:48:21.201351 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.201323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-dshm" (OuterVolumeSpecName: "dshm") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:21.201470 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.201436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:48:21.225060 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.225022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f83faef-82c8-46b5-8a35-2c372f87c3f1" (UID: "9f83faef-82c8-46b5-8a35-2c372f87c3f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:48:21.300022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.299937    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f83faef-82c8-46b5-8a35-2c372f87c3f1-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.300022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.299969    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.300022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.299977    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f83faef-82c8-46b5-8a35-2c372f87c3f1-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.300022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.299985    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.300022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.299994    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pkdg\" (UniqueName: \"kubernetes.io/projected/9f83faef-82c8-46b5-8a35-2c372f87c3f1-kube-api-access-5pkdg\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.300022 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.300003    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f83faef-82c8-46b5-8a35-2c372f87c3f1-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:48:21.373669 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.373641    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"]
Sep 19 08:48:21.377673 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:21.377646    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-vcqqs"]
Sep 19 08:48:22.080093 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:22.080058    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" path="/var/lib/kubelet/pods/9f83faef-82c8-46b5-8a35-2c372f87c3f1/volumes"
Sep 19 08:48:29.722390 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:29.722350    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:48:30.631757 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:30.631710    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main" containerID="cri-o://74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087" gracePeriod=45
Sep 19 08:48:37.137813 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:37.137779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" event={"ID":"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe","Type":"ContainerStarted","Data":"2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137"}
Sep 19 08:48:37.161457 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:37.161397    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podStartSLOduration=2.564244169 podStartE2EDuration="32.161380869s" totalImagesPullingTime="27.373570863s" totalInitContainerRuntime="2.223565837s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:48:37.158970573 +0000 UTC m=+1669.345162555" watchObservedRunningTime="2026-09-19 08:48:37.161380869 +0000 UTC m=+1669.347572826"
Sep 19 08:48:37.313390 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:37.313355    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:38.965812 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:38.965784    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:48:39.722556 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:39.722516    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:48:39.722731 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:39.722617    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:48:46.312684 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:46.312648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:48:46.314029 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:46.314000    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:48:49.722806 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:49.722765    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:48:56.312892 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:56.312847    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:48:59.722999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:48:59.722963    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:49:00.876914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:00.876883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:49:00.956945 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:00.956903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svbwf\" (UniqueName: \"kubernetes.io/projected/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-kube-api-access-svbwf\") pod \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") "
Sep 19 08:49:00.957128 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:00.956959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-tls-certs\") pod \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\" (UID: \"c56cc31c-52a0-4f2c-b2da-66d1dba58f02\") "
Sep 19 08:49:00.958971 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:00.958947    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c56cc31c-52a0-4f2c-b2da-66d1dba58f02" (UID: "c56cc31c-52a0-4f2c-b2da-66d1dba58f02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:49:00.959065 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:00.959041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-kube-api-access-svbwf" (OuterVolumeSpecName: "kube-api-access-svbwf") pod "c56cc31c-52a0-4f2c-b2da-66d1dba58f02" (UID: "c56cc31c-52a0-4f2c-b2da-66d1dba58f02"). InnerVolumeSpecName "kube-api-access-svbwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:49:01.057966 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.057932    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svbwf\" (UniqueName: \"kubernetes.io/projected/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-kube-api-access-svbwf\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:01.057966 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.057960    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c56cc31c-52a0-4f2c-b2da-66d1dba58f02-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:01.246820 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.246728    2580 generic.go:356] "Generic (PLEG): container finished" podID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerID="74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087" exitCode=0
Sep 19 08:49:01.246820 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.246772    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"
Sep 19 08:49:01.246820 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.246813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" event={"ID":"c56cc31c-52a0-4f2c-b2da-66d1dba58f02","Type":"ContainerDied","Data":"74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087"}
Sep 19 08:49:01.247084 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.246864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76" event={"ID":"c56cc31c-52a0-4f2c-b2da-66d1dba58f02","Type":"ContainerDied","Data":"a1d2624cb2f6fbeb9713c49c56cec1ec97fe08ed5446d5368a9ea08c8d324a9f"}
Sep 19 08:49:01.247084 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.246887    2580 scope.go:117] "RemoveContainer" containerID="74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087"
Sep 19 08:49:01.255964 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.255947    2580 scope.go:117] "RemoveContainer" containerID="74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087"
Sep 19 08:49:01.256344 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:49:01.256320    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087\": container with ID starting with 74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087 not found: ID does not exist" containerID="74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087"
Sep 19 08:49:01.256447 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.256353    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087"} err="failed to get container status \"74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087\": rpc error: code = NotFound desc = could not find container \"74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087\": container with ID starting with 74f8ce2a1370d7935d63f86a37d1879e895de59437b459277495b2e70de2f087 not found: ID does not exist"
Sep 19 08:49:01.269420 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.269390    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"]
Sep 19 08:49:01.271429 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:01.271405    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785kwm76"]
Sep 19 08:49:02.079673 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:02.079641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" path="/var/lib/kubelet/pods/c56cc31c-52a0-4f2c-b2da-66d1dba58f02/volumes"
Sep 19 08:49:06.313116 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:06.313070    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:49:11.293506 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293473    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"]
Sep 19 08:49:11.293881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293789    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="storage-initializer"
Sep 19 08:49:11.293881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293800    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="storage-initializer"
Sep 19 08:49:11.293881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293808    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main"
Sep 19 08:49:11.293881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main"
Sep 19 08:49:11.293881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293824    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main"
Sep 19 08:49:11.293881 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293829    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main"
Sep 19 08:49:11.294075 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293886    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56cc31c-52a0-4f2c-b2da-66d1dba58f02" containerName="main"
Sep 19 08:49:11.294075 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.293898    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f83faef-82c8-46b5-8a35-2c372f87c3f1" containerName="main"
Sep 19 08:49:11.299465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.299447    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.302507 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.302479    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 19 08:49:11.303300 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.303278    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:49:11.303469 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.303297    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 19 08:49:11.303545 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.303357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 19 08:49:11.303597 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.303361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 19 08:49:11.303597 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.303373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-wlb6m\""
Sep 19 08:49:11.309024 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.309001    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"]
Sep 19 08:49:11.449407 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449374    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlzv5\" (UniqueName: \"kubernetes.io/projected/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kube-api-access-tlzv5\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.449805 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.449672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlzv5\" (UniqueName: \"kubernetes.io/projected/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kube-api-access-tlzv5\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550747 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550747 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550567    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550747 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550747 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550631    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550963 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.550963 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.550879    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.551115 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.551094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.551179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.551118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.551262 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.551197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.553146 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.553108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.553294 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.553281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.563985 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.563958    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"]
Sep 19 08:49:11.564524 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.564501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlzv5\" (UniqueName: \"kubernetes.io/projected/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kube-api-access-tlzv5\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-zhr95\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.575975 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.575954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.577689 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.577666    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"]
Sep 19 08:49:11.578241 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.578204    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-gxvls\""
Sep 19 08:49:11.611160 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.611126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:11.752563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.752528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb4f713-a337-4fed-a51b-e34cda97785d-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.752748 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.752598    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgksx\" (UniqueName: \"kubernetes.io/projected/efb4f713-a337-4fed-a51b-e34cda97785d-kube-api-access-mgksx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.853990 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.853890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb4f713-a337-4fed-a51b-e34cda97785d-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.854153 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.854003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgksx\" (UniqueName: \"kubernetes.io/projected/efb4f713-a337-4fed-a51b-e34cda97785d-kube-api-access-mgksx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.856451 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.856421    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb4f713-a337-4fed-a51b-e34cda97785d-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.862684 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.862654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgksx\" (UniqueName: \"kubernetes.io/projected/efb4f713-a337-4fed-a51b-e34cda97785d-kube-api-access-mgksx\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:11.885521 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.885495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"]
Sep 19 08:49:11.888535 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:49:11.888510    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e1364e_5cb2_45d8_817a_d0a87a4b27ca.slice/crio-739cff29cdc6fb97b3e32e64cc6a62744673ad3b6e2cf18000ace42e79edcbaa WatchSource:0}: Error finding container 739cff29cdc6fb97b3e32e64cc6a62744673ad3b6e2cf18000ace42e79edcbaa: Status 404 returned error can't find the container with id 739cff29cdc6fb97b3e32e64cc6a62744673ad3b6e2cf18000ace42e79edcbaa
Sep 19 08:49:11.890464 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:11.890445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:12.168809 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.168735    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"]
Sep 19 08:49:12.171350 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:49:12.171324    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb4f713_a337_4fed_a51b_e34cda97785d.slice/crio-c76dc1d4999b63df7233df1f4aa1b864892b3aa4a8f396aa6830c73e43e8f4c0 WatchSource:0}: Error finding container c76dc1d4999b63df7233df1f4aa1b864892b3aa4a8f396aa6830c73e43e8f4c0: Status 404 returned error can't find the container with id c76dc1d4999b63df7233df1f4aa1b864892b3aa4a8f396aa6830c73e43e8f4c0
Sep 19 08:49:12.299989 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.299940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" event={"ID":"efb4f713-a337-4fed-a51b-e34cda97785d","Type":"ContainerStarted","Data":"9237889be44b850a00f8c920ccecfbf2bf2dd58e439bc02b042aef0a8b3fa11d"}
Sep 19 08:49:12.300500 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.300014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:12.300500 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.300044    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" event={"ID":"efb4f713-a337-4fed-a51b-e34cda97785d","Type":"ContainerStarted","Data":"c76dc1d4999b63df7233df1f4aa1b864892b3aa4a8f396aa6830c73e43e8f4c0"}
Sep 19 08:49:12.302589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.302003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" event={"ID":"86e1364e-5cb2-45d8-817a-d0a87a4b27ca","Type":"ContainerStarted","Data":"4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7"}
Sep 19 08:49:12.302589 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.302059    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" event={"ID":"86e1364e-5cb2-45d8-817a-d0a87a4b27ca","Type":"ContainerStarted","Data":"739cff29cdc6fb97b3e32e64cc6a62744673ad3b6e2cf18000ace42e79edcbaa"}
Sep 19 08:49:12.320902 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:12.320845    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podStartSLOduration=1.320822498 podStartE2EDuration="1.320822498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:12.31793832 +0000 UTC m=+1704.504130277" watchObservedRunningTime="2026-09-19 08:49:12.320822498 +0000 UTC m=+1704.507014457"
Sep 19 08:49:15.317595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:15.317556    2580 generic.go:356] "Generic (PLEG): container finished" podID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerID="4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7" exitCode=0
Sep 19 08:49:15.317595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:15.317584    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" event={"ID":"86e1364e-5cb2-45d8-817a-d0a87a4b27ca","Type":"ContainerDied","Data":"4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7"}
Sep 19 08:49:16.313446 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:16.313406    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:49:16.324234 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:16.324189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" event={"ID":"86e1364e-5cb2-45d8-817a-d0a87a4b27ca","Type":"ContainerStarted","Data":"78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"}
Sep 19 08:49:16.343599 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:16.343551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" podStartSLOduration=2.905090112 podStartE2EDuration="5.343536s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.438445888s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:49:16.342956193 +0000 UTC m=+1708.529148164" watchObservedRunningTime="2026-09-19 08:49:16.343536 +0000 UTC m=+1708.529727957"
Sep 19 08:49:16.612387 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:16.612307    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:21.611556 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:21.611513    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:21.624487 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:21.624461    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:22.364012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:22.363983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:26.314452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:26.314156    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:49:36.313009 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:36.312966    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:49:43.308307 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:43.308273    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:49:46.313445 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:46.313399    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:49:49.430777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.430739    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"]
Sep 19 08:49:49.434327 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.434294    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"]
Sep 19 08:49:49.444525 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:49:49.444481    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:49:49.444525 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:49:49.444525 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:49:49.444525 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db" cmd=["/bin/sleep","15"]
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.444546    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" message="" err=<
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:49:49.444563    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="main" containerID="cri-o://78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"
Sep 19 08:49:49.444828 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.444607    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="main" containerID="cri-o://78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db" gracePeriod=60
Sep 19 08:49:49.801026 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.800995    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:49.902011 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.901978    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tls-certs\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-dshm\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-cabundle-cert\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902084    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlzv5\" (UniqueName: \"kubernetes.io/projected/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kube-api-access-tlzv5\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-model-cache\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tmp-dir\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kserve-provision-location\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902306    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-home\") pod \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\" (UID: \"86e1364e-5cb2-45d8-817a-d0a87a4b27ca\") "
Sep 19 08:49:49.902496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902453    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:49:49.902496 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.902700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.902700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-home" (OuterVolumeSpecName: "home") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.902700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902657    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.902700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902677    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.902700 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.902691    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:49.904563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.904525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kube-api-access-tlzv5" (OuterVolumeSpecName: "kube-api-access-tlzv5") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "kube-api-access-tlzv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:49:49.904563 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.904537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:49:49.904752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.904705    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-dshm" (OuterVolumeSpecName: "dshm") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:49.922456 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:49.922421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "86e1364e-5cb2-45d8-817a-d0a87a4b27ca" (UID: "86e1364e-5cb2-45d8-817a-d0a87a4b27ca"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:49:50.003680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.003594    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.003680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.003625    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlzv5\" (UniqueName: \"kubernetes.io/projected/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kube-api-access-tlzv5\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.003680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.003638    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.003680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.003651    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.003680 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.003662    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e1364e-5cb2-45d8-817a-d0a87a4b27ca-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:49:50.493834 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.493802    2580 generic.go:356] "Generic (PLEG): container finished" podID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerID="78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db" exitCode=0
Sep 19 08:49:50.494280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.493856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" event={"ID":"86e1364e-5cb2-45d8-817a-d0a87a4b27ca","Type":"ContainerDied","Data":"78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"}
Sep 19 08:49:50.494280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.493866    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"
Sep 19 08:49:50.494280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.493920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95" event={"ID":"86e1364e-5cb2-45d8-817a-d0a87a4b27ca","Type":"ContainerDied","Data":"739cff29cdc6fb97b3e32e64cc6a62744673ad3b6e2cf18000ace42e79edcbaa"}
Sep 19 08:49:50.494280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.493944    2580 scope.go:117] "RemoveContainer" containerID="78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"
Sep 19 08:49:50.502444 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.502426    2580 scope.go:117] "RemoveContainer" containerID="4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7"
Sep 19 08:49:50.509805 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.509780    2580 scope.go:117] "RemoveContainer" containerID="78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"
Sep 19 08:49:50.510472 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:49:50.510094    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db\": container with ID starting with 78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db not found: ID does not exist" containerID="78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"
Sep 19 08:49:50.510472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.510131    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db"} err="failed to get container status \"78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db\": rpc error: code = NotFound desc = could not find container \"78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db\": container with ID starting with 78e68430fb932f37770c1e989985ebfc2c14c6dcd4609b7d79fac48c431479db not found: ID does not exist"
Sep 19 08:49:50.510472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.510154    2580 scope.go:117] "RemoveContainer" containerID="4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7"
Sep 19 08:49:50.510676 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:49:50.510473    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7\": container with ID starting with 4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7 not found: ID does not exist" containerID="4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7"
Sep 19 08:49:50.510676 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.510500    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7"} err="failed to get container status \"4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7\": rpc error: code = NotFound desc = could not find container \"4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7\": container with ID starting with 4e2797b2192539959c7e685b321c1a2e050b41dfbd8f741c1ef4512942dfd0a7 not found: ID does not exist"
Sep 19 08:49:50.511633 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.511615    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"]
Sep 19 08:49:50.518109 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:50.518089    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-zhr95"]
Sep 19 08:49:52.080775 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:52.080729    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" path="/var/lib/kubelet/pods/86e1364e-5cb2-45d8-817a-d0a87a4b27ca/volumes"
Sep 19 08:49:53.308944 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:53.308897    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:49:56.313198 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:49:56.313140    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:50:03.309308 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:03.309268    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:04.457967 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:04.457923    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main" containerID="cri-o://9237889be44b850a00f8c920ccecfbf2bf2dd58e439bc02b042aef0a8b3fa11d" gracePeriod=45
Sep 19 08:50:06.312824 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:06.312774    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:50:13.308846 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:13.308809    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:13.309277 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:13.308929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:50:16.313044 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:16.312998    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:50:23.308383 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:23.308339    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:26.313265 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:26.313200    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:50:33.308368 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:33.308269    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:34.702327 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.702299    2580 generic.go:356] "Generic (PLEG): container finished" podID="efb4f713-a337-4fed-a51b-e34cda97785d" containerID="9237889be44b850a00f8c920ccecfbf2bf2dd58e439bc02b042aef0a8b3fa11d" exitCode=0
Sep 19 08:50:34.702667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.702345    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" event={"ID":"efb4f713-a337-4fed-a51b-e34cda97785d","Type":"ContainerDied","Data":"9237889be44b850a00f8c920ccecfbf2bf2dd58e439bc02b042aef0a8b3fa11d"}
Sep 19 08:50:34.702667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.702385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb" event={"ID":"efb4f713-a337-4fed-a51b-e34cda97785d","Type":"ContainerDied","Data":"c76dc1d4999b63df7233df1f4aa1b864892b3aa4a8f396aa6830c73e43e8f4c0"}
Sep 19 08:50:34.702667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.702403    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76dc1d4999b63df7233df1f4aa1b864892b3aa4a8f396aa6830c73e43e8f4c0"
Sep 19 08:50:34.702667 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.702525    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:50:34.805754 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.805673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgksx\" (UniqueName: \"kubernetes.io/projected/efb4f713-a337-4fed-a51b-e34cda97785d-kube-api-access-mgksx\") pod \"efb4f713-a337-4fed-a51b-e34cda97785d\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") "
Sep 19 08:50:34.805900 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.805757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb4f713-a337-4fed-a51b-e34cda97785d-tls-certs\") pod \"efb4f713-a337-4fed-a51b-e34cda97785d\" (UID: \"efb4f713-a337-4fed-a51b-e34cda97785d\") "
Sep 19 08:50:34.807808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.807772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb4f713-a337-4fed-a51b-e34cda97785d-kube-api-access-mgksx" (OuterVolumeSpecName: "kube-api-access-mgksx") pod "efb4f713-a337-4fed-a51b-e34cda97785d" (UID: "efb4f713-a337-4fed-a51b-e34cda97785d"). InnerVolumeSpecName "kube-api-access-mgksx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:50:34.807928 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.807779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb4f713-a337-4fed-a51b-e34cda97785d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efb4f713-a337-4fed-a51b-e34cda97785d" (UID: "efb4f713-a337-4fed-a51b-e34cda97785d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:50:34.906414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.906379    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgksx\" (UniqueName: \"kubernetes.io/projected/efb4f713-a337-4fed-a51b-e34cda97785d-kube-api-access-mgksx\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:34.906414 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:34.906408    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efb4f713-a337-4fed-a51b-e34cda97785d-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:35.706969 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:35.706938    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"
Sep 19 08:50:35.727056 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:35.727028    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"]
Sep 19 08:50:35.733194 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:35.733172    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dp74jb"]
Sep 19 08:50:36.080182 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:36.080101    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" path="/var/lib/kubelet/pods/efb4f713-a337-4fed-a51b-e34cda97785d/volumes"
Sep 19 08:50:36.323382 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:36.323346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:50:36.331752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:36.331676    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:50:37.692749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:37.692719    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"]
Sep 19 08:50:37.708419 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:37.708387    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"]
Sep 19 08:50:38.966189 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:38.966153    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:48.966289 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:48.966250    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:52.718364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:52.718308    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main" containerID="cri-o://4f4c3d6a230a2ac751f06ac6ed542e972c81bc9f067759eb6dcd783d85ecd2c3" gracePeriod=45
Sep 19 08:50:52.738564 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:52.738529    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" containerID="cri-o://2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137" gracePeriod=45
Sep 19 08:50:53.324165 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.324122    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 19 08:50:53.486398 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.486374    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:50:53.561686 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561607    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-home\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.561686 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tls-certs\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.561686 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-cabundle-cert\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.561686 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561689    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-model-cache\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.562005 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tmp-dir\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.562005 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561755    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-dshm\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.562005 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxs7p\" (UniqueName: \"kubernetes.io/projected/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kube-api-access-xxs7p\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.562005 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.561820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kserve-provision-location\") pod \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\" (UID: \"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe\") "
Sep 19 08:50:53.562192 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.562040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:50:53.562361 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.562332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:53.562448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.562409    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-home" (OuterVolumeSpecName: "home") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:53.564173 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.564137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kube-api-access-xxs7p" (OuterVolumeSpecName: "kube-api-access-xxs7p") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "kube-api-access-xxs7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:50:53.564173 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.564150    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:50:53.564406 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.564172    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-dshm" (OuterVolumeSpecName: "dshm") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:53.581101 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.581073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:53.589453 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.589432    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" (UID: "2dd1dbb1-ee12-415b-aeda-5032c1f35cbe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:50:53.662712 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662692    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662714    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662723    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662732    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662739    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662746    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662754    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxs7p\" (UniqueName: \"kubernetes.io/projected/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kube-api-access-xxs7p\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.662799 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.662762    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:50:53.785245 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.785197    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerID="2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137" exitCode=0
Sep 19 08:50:53.785585 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.785271    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"
Sep 19 08:50:53.785585 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.785267    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" event={"ID":"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe","Type":"ContainerDied","Data":"2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137"}
Sep 19 08:50:53.785585 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.785393    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr" event={"ID":"2dd1dbb1-ee12-415b-aeda-5032c1f35cbe","Type":"ContainerDied","Data":"86b6bd87627c30e406a9589f9bf3cd11c08de1a9ab9334bea867b37060a7b3bd"}
Sep 19 08:50:53.785585 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.785422    2580 scope.go:117] "RemoveContainer" containerID="2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137"
Sep 19 08:50:53.794275 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.794256    2580 scope.go:117] "RemoveContainer" containerID="c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c"
Sep 19 08:50:53.802131 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.802111    2580 scope.go:117] "RemoveContainer" containerID="2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137"
Sep 19 08:50:53.802402 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:50:53.802382    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137\": container with ID starting with 2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137 not found: ID does not exist" containerID="2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137"
Sep 19 08:50:53.802480 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.802414    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137"} err="failed to get container status \"2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137\": rpc error: code = NotFound desc = could not find container \"2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137\": container with ID starting with 2bcd2b5123d0e8c1e5b4af658c4dd77c0bdfa85936e2d5a4e8110dfd56765137 not found: ID does not exist"
Sep 19 08:50:53.802480 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.802436    2580 scope.go:117] "RemoveContainer" containerID="c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c"
Sep 19 08:50:53.802703 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:50:53.802683    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c\": container with ID starting with c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c not found: ID does not exist" containerID="c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c"
Sep 19 08:50:53.802746 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.802709    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c"} err="failed to get container status \"c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c\": rpc error: code = NotFound desc = could not find container \"c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c\": container with ID starting with c394b2a91ba3eeb934f34db8f30cb49c6ac8e2fdbaa5ec841161813042070d8c not found: ID does not exist"
Sep 19 08:50:53.807503 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.807481    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"]
Sep 19 08:50:53.812214 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:53.812164    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-rttgr"]
Sep 19 08:50:54.079527 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:54.079448    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" path="/var/lib/kubelet/pods/2dd1dbb1-ee12-415b-aeda-5032c1f35cbe/volumes"
Sep 19 08:50:58.965807 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:58.965769    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:50:58.966322 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:50:58.965896    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:51:08.966271 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:08.966200    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:51:18.965609 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:18.965569    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:51:22.915268 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:22.915203    2580 generic.go:356] "Generic (PLEG): container finished" podID="f89b6ed9-9998-4015-b719-c082cb132e06" containerID="4f4c3d6a230a2ac751f06ac6ed542e972c81bc9f067759eb6dcd783d85ecd2c3" exitCode=0
Sep 19 08:51:22.915608 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:22.915284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" event={"ID":"f89b6ed9-9998-4015-b719-c082cb132e06","Type":"ContainerDied","Data":"4f4c3d6a230a2ac751f06ac6ed542e972c81bc9f067759eb6dcd783d85ecd2c3"}
Sep 19 08:51:22.956648 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:22.956626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:51:23.107984 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.107899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89b6ed9-9998-4015-b719-c082cb132e06-tls-certs\") pod \"f89b6ed9-9998-4015-b719-c082cb132e06\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") "
Sep 19 08:51:23.107984 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.107951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pr76\" (UniqueName: \"kubernetes.io/projected/f89b6ed9-9998-4015-b719-c082cb132e06-kube-api-access-8pr76\") pod \"f89b6ed9-9998-4015-b719-c082cb132e06\" (UID: \"f89b6ed9-9998-4015-b719-c082cb132e06\") "
Sep 19 08:51:23.110058 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.110021    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89b6ed9-9998-4015-b719-c082cb132e06-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f89b6ed9-9998-4015-b719-c082cb132e06" (UID: "f89b6ed9-9998-4015-b719-c082cb132e06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:51:23.110169 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.110075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b6ed9-9998-4015-b719-c082cb132e06-kube-api-access-8pr76" (OuterVolumeSpecName: "kube-api-access-8pr76") pod "f89b6ed9-9998-4015-b719-c082cb132e06" (UID: "f89b6ed9-9998-4015-b719-c082cb132e06"). InnerVolumeSpecName "kube-api-access-8pr76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:51:23.208897 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.208863    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89b6ed9-9998-4015-b719-c082cb132e06-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.208897 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.208891    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pr76\" (UniqueName: \"kubernetes.io/projected/f89b6ed9-9998-4015-b719-c082cb132e06-kube-api-access-8pr76\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:51:23.921727 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.921687    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"
Sep 19 08:51:23.921727 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.921712    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks" event={"ID":"f89b6ed9-9998-4015-b719-c082cb132e06","Type":"ContainerDied","Data":"56ba7e433ce1e33d783a03b6e752f8ed2043350e4b3f7003403fa817a4947aa8"}
Sep 19 08:51:23.922332 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.921777    2580 scope.go:117] "RemoveContainer" containerID="4f4c3d6a230a2ac751f06ac6ed542e972c81bc9f067759eb6dcd783d85ecd2c3"
Sep 19 08:51:23.944829 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.944797    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"]
Sep 19 08:51:23.953357 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:23.948544    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cr7cks"]
Sep 19 08:51:24.078513 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:24.078473    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" path="/var/lib/kubelet/pods/f89b6ed9-9998-4015-b719-c082cb132e06/volumes"
Sep 19 08:51:35.061952 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.061914    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"]
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062239    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062254    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062262    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062268    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062275    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="storage-initializer"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062280    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="storage-initializer"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062294    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062299    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062311    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="storage-initializer"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062317    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="storage-initializer"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062327    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="main"
Sep 19 08:51:35.062364 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062334    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="main"
Sep 19 08:51:35.062749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062388    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd1dbb1-ee12-415b-aeda-5032c1f35cbe" containerName="main"
Sep 19 08:51:35.062749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062398    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="efb4f713-a337-4fed-a51b-e34cda97785d" containerName="main"
Sep 19 08:51:35.062749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062409    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86e1364e-5cb2-45d8-817a-d0a87a4b27ca" containerName="main"
Sep 19 08:51:35.062749 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.062417    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f89b6ed9-9998-4015-b719-c082cb132e06" containerName="main"
Sep 19 08:51:35.066944 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.066924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.069506 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.069485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-cpl94\""
Sep 19 08:51:35.069623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.069552    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 19 08:51:35.069623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.069554    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 19 08:51:35.069623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.069591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 19 08:51:35.069623 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.069597    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:51:35.069838 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.069569    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 19 08:51:35.078100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.078077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"]
Sep 19 08:51:35.107895 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.107864    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-model-cache\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.107905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzk2\" (UniqueName: \"kubernetes.io/projected/47944175-0eb0-4181-924c-826a46737bb9-kube-api-access-rdzk2\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.107934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-dshm\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.107948    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47944175-0eb0-4181-924c-826a46737bb9-cabundle-cert\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.107995    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-tmp-dir\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.108010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-home\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.108034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47944175-0eb0-4181-924c-826a46737bb9-tls-certs\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.108242 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.108055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-kserve-provision-location\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209359 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-model-cache\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209366    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzk2\" (UniqueName: \"kubernetes.io/projected/47944175-0eb0-4181-924c-826a46737bb9-kube-api-access-rdzk2\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209397    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-dshm\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47944175-0eb0-4181-924c-826a46737bb9-cabundle-cert\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-tmp-dir\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-home\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47944175-0eb0-4181-924c-826a46737bb9-tls-certs\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209879 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-kserve-provision-location\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-home\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.209943 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-tmp-dir\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.210050 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.209991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-kserve-provision-location\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.210110 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.210098    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-model-cache\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.210161 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.210132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47944175-0eb0-4181-924c-826a46737bb9-cabundle-cert\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.211790 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.211770    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-dshm\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.212117 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.212099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47944175-0eb0-4181-924c-826a46737bb9-tls-certs\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.218020 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.217998    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzk2\" (UniqueName: \"kubernetes.io/projected/47944175-0eb0-4181-924c-826a46737bb9-kube-api-access-rdzk2\") pod \"auth-disabled-test-kserve-54f6b74bc-jg2pk\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.322963 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.322877    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"]
Sep 19 08:51:35.327230 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.327204    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.329909 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.329885    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-xz9pf\""
Sep 19 08:51:35.336087 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.336061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"]
Sep 19 08:51:35.378130 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.378097    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:35.411411 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.411373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjtl\" (UniqueName: \"kubernetes.io/projected/9bf6188c-b05a-442e-bca7-caf59eb22f7c-kube-api-access-5bjtl\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.411555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.411458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6188c-b05a-442e-bca7-caf59eb22f7c-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.512752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.512720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjtl\" (UniqueName: \"kubernetes.io/projected/9bf6188c-b05a-442e-bca7-caf59eb22f7c-kube-api-access-5bjtl\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.512909 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.512773    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6188c-b05a-442e-bca7-caf59eb22f7c-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.515154 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.515136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6188c-b05a-442e-bca7-caf59eb22f7c-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.520428 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.520404    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjtl\" (UniqueName: \"kubernetes.io/projected/9bf6188c-b05a-442e-bca7-caf59eb22f7c-kube-api-access-5bjtl\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.638052 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.637976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:35.662391 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.660392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"]
Sep 19 08:51:35.663488 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:51:35.663444    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47944175_0eb0_4181_924c_826a46737bb9.slice/crio-e712e05ed721e97b2f1f13cfe24d3cda5a5cef88714c4b9dcf91ab52bc2e1deb WatchSource:0}: Error finding container e712e05ed721e97b2f1f13cfe24d3cda5a5cef88714c4b9dcf91ab52bc2e1deb: Status 404 returned error can't find the container with id e712e05ed721e97b2f1f13cfe24d3cda5a5cef88714c4b9dcf91ab52bc2e1deb
Sep 19 08:51:35.915810 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.915784    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"]
Sep 19 08:51:35.918193 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:51:35.918160    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf6188c_b05a_442e_bca7_caf59eb22f7c.slice/crio-2f54f0dfd0c0e5a5ae4d6135f10a8707e0b5193a54b4a027664e108d49631797 WatchSource:0}: Error finding container 2f54f0dfd0c0e5a5ae4d6135f10a8707e0b5193a54b4a027664e108d49631797: Status 404 returned error can't find the container with id 2f54f0dfd0c0e5a5ae4d6135f10a8707e0b5193a54b4a027664e108d49631797
Sep 19 08:51:35.976626 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.976594    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" event={"ID":"9bf6188c-b05a-442e-bca7-caf59eb22f7c","Type":"ContainerStarted","Data":"2f54f0dfd0c0e5a5ae4d6135f10a8707e0b5193a54b4a027664e108d49631797"}
Sep 19 08:51:35.978494 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.978464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" event={"ID":"47944175-0eb0-4181-924c-826a46737bb9","Type":"ContainerStarted","Data":"8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300"}
Sep 19 08:51:35.978613 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:35.978532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" event={"ID":"47944175-0eb0-4181-924c-826a46737bb9","Type":"ContainerStarted","Data":"e712e05ed721e97b2f1f13cfe24d3cda5a5cef88714c4b9dcf91ab52bc2e1deb"}
Sep 19 08:51:36.984710 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:36.984663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" event={"ID":"9bf6188c-b05a-442e-bca7-caf59eb22f7c","Type":"ContainerStarted","Data":"de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223"}
Sep 19 08:51:36.985195 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:36.984767    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:51:37.001674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:37.001629    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" podStartSLOduration=2.001615665 podStartE2EDuration="2.001615665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:37.000118936 +0000 UTC m=+1849.186310936" watchObservedRunningTime="2026-09-19 08:51:37.001615665 +0000 UTC m=+1849.187807621"
Sep 19 08:51:37.991166 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:37.991128    2580 generic.go:356] "Generic (PLEG): container finished" podID="47944175-0eb0-4181-924c-826a46737bb9" containerID="8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300" exitCode=0
Sep 19 08:51:37.991516 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:37.991205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" event={"ID":"47944175-0eb0-4181-924c-826a46737bb9","Type":"ContainerDied","Data":"8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300"}
Sep 19 08:51:38.998042 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:38.998008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" event={"ID":"47944175-0eb0-4181-924c-826a46737bb9","Type":"ContainerStarted","Data":"474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab"}
Sep 19 08:51:39.021211 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:39.021155    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podStartSLOduration=1.827117888 podStartE2EDuration="4.021139565s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.194021677s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:51:39.018501796 +0000 UTC m=+1851.204693764" watchObservedRunningTime="2026-09-19 08:51:39.021139565 +0000 UTC m=+1851.207331522"
Sep 19 08:51:39.378576 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:39.378494    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:45.379162 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:45.379125    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:51:45.380395 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:45.380369    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:51:55.379110 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:51:55.379068    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:52:05.379055 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:05.378946    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:52:07.993124 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:07.993096    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:52:15.379216 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:15.379170    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:52:25.379072 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:25.379023    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:52:35.378959 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:35.378913    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:52:45.379448 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:45.379405    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:52:55.379010 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:52:55.378965    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:53:05.378772 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:05.378734    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:53:15.378615 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:15.378570    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:53:25.379540 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:25.379497    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:53:35.388988 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:35.388892    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:53:35.397782 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:35.397755    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:53:40.208300 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:40.208262    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"]
Sep 19 08:53:40.218210 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:40.218176    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"]
Sep 19 08:53:47.992986 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:47.992948    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:53:55.230701 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:55.230643    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main" containerID="cri-o://de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223" gracePeriod=45
Sep 19 08:53:55.242282 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:55.242248    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" containerID="cri-o://474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab" gracePeriod=45
Sep 19 08:53:55.390190 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:55.390143    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 19 08:53:55.982298 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:55.982275    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:53:56.105314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-tmp-dir\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105292    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdzk2\" (UniqueName: \"kubernetes.io/projected/47944175-0eb0-4181-924c-826a46737bb9-kube-api-access-rdzk2\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105314 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-model-cache\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47944175-0eb0-4181-924c-826a46737bb9-cabundle-cert\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-dshm\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-home\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105454    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-kserve-provision-location\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105602 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47944175-0eb0-4181-924c-826a46737bb9-tls-certs\") pod \"47944175-0eb0-4181-924c-826a46737bb9\" (UID: \"47944175-0eb0-4181-924c-826a46737bb9\") "
Sep 19 08:53:56.105875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:56.105875 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.105808    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.106152 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.106121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47944175-0eb0-4181-924c-826a46737bb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:53:56.106277 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.106209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-home" (OuterVolumeSpecName: "home") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:56.108197 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.108159    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-dshm" (OuterVolumeSpecName: "dshm") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:56.108299 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.108196    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47944175-0eb0-4181-924c-826a46737bb9-kube-api-access-rdzk2" (OuterVolumeSpecName: "kube-api-access-rdzk2") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "kube-api-access-rdzk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:53:56.109477 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.109456    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47944175-0eb0-4181-924c-826a46737bb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:53:56.124321 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.124285    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:56.135120 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.135099    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47944175-0eb0-4181-924c-826a46737bb9" (UID: "47944175-0eb0-4181-924c-826a46737bb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:53:56.207140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207108    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.207140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207140    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47944175-0eb0-4181-924c-826a46737bb9-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.207350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207153    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.207350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207166    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzk2\" (UniqueName: \"kubernetes.io/projected/47944175-0eb0-4181-924c-826a46737bb9-kube-api-access-rdzk2\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.207350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207178    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47944175-0eb0-4181-924c-826a46737bb9-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.207350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207189    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.207350 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.207199    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47944175-0eb0-4181-924c-826a46737bb9-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:53:56.619100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.619058    2580 generic.go:356] "Generic (PLEG): container finished" podID="47944175-0eb0-4181-924c-826a46737bb9" containerID="474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab" exitCode=0
Sep 19 08:53:56.619555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.619111    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"
Sep 19 08:53:56.619555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.619117    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" event={"ID":"47944175-0eb0-4181-924c-826a46737bb9","Type":"ContainerDied","Data":"474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab"}
Sep 19 08:53:56.619555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.619172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk" event={"ID":"47944175-0eb0-4181-924c-826a46737bb9","Type":"ContainerDied","Data":"e712e05ed721e97b2f1f13cfe24d3cda5a5cef88714c4b9dcf91ab52bc2e1deb"}
Sep 19 08:53:56.619555 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.619194    2580 scope.go:117] "RemoveContainer" containerID="474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab"
Sep 19 08:53:56.628515 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.628496    2580 scope.go:117] "RemoveContainer" containerID="8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300"
Sep 19 08:53:56.635999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.635979    2580 scope.go:117] "RemoveContainer" containerID="474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab"
Sep 19 08:53:56.636287 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:53:56.636263    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab\": container with ID starting with 474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab not found: ID does not exist" containerID="474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab"
Sep 19 08:53:56.636345 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.636302    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab"} err="failed to get container status \"474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab\": rpc error: code = NotFound desc = could not find container \"474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab\": container with ID starting with 474a779fdd3621bf36a94eb75ec537ed5491f475117324a4f1c19c976bec43ab not found: ID does not exist"
Sep 19 08:53:56.636345 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.636326    2580 scope.go:117] "RemoveContainer" containerID="8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300"
Sep 19 08:53:56.636614 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:53:56.636595    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300\": container with ID starting with 8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300 not found: ID does not exist" containerID="8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300"
Sep 19 08:53:56.636674 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.636619    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300"} err="failed to get container status \"8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300\": rpc error: code = NotFound desc = could not find container \"8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300\": container with ID starting with 8a774ffa19c6376a4772161dfc02b055fb063228b269c62ea3b37c475e331300 not found: ID does not exist"
Sep 19 08:53:56.641497 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.641476    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"]
Sep 19 08:53:56.644948 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:56.644922    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-jg2pk"]
Sep 19 08:53:57.993080 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:57.993039    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:53:58.079407 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:53:58.079375    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47944175-0eb0-4181-924c-826a46737bb9" path="/var/lib/kubelet/pods/47944175-0eb0-4181-924c-826a46737bb9/volumes"
Sep 19 08:54:07.992944 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:07.992897    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:54:07.993382 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:07.993008    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:54:17.993924 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:17.993885    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:54:25.471511 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.471484    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:54:25.547638 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.547566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6188c-b05a-442e-bca7-caf59eb22f7c-tls-certs\") pod \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") "
Sep 19 08:54:25.547638 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.547633    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bjtl\" (UniqueName: \"kubernetes.io/projected/9bf6188c-b05a-442e-bca7-caf59eb22f7c-kube-api-access-5bjtl\") pod \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\" (UID: \"9bf6188c-b05a-442e-bca7-caf59eb22f7c\") "
Sep 19 08:54:25.549653 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.549618    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf6188c-b05a-442e-bca7-caf59eb22f7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bf6188c-b05a-442e-bca7-caf59eb22f7c" (UID: "9bf6188c-b05a-442e-bca7-caf59eb22f7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:54:25.549761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.549700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf6188c-b05a-442e-bca7-caf59eb22f7c-kube-api-access-5bjtl" (OuterVolumeSpecName: "kube-api-access-5bjtl") pod "9bf6188c-b05a-442e-bca7-caf59eb22f7c" (UID: "9bf6188c-b05a-442e-bca7-caf59eb22f7c"). InnerVolumeSpecName "kube-api-access-5bjtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:54:25.648958 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.648913    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bf6188c-b05a-442e-bca7-caf59eb22f7c-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:54:25.648958 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.648952    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjtl\" (UniqueName: \"kubernetes.io/projected/9bf6188c-b05a-442e-bca7-caf59eb22f7c-kube-api-access-5bjtl\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:54:25.745307 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.745275    2580 generic.go:356] "Generic (PLEG): container finished" podID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerID="de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223" exitCode=0
Sep 19 08:54:25.745465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.745321    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"
Sep 19 08:54:25.745465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.745322    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" event={"ID":"9bf6188c-b05a-442e-bca7-caf59eb22f7c","Type":"ContainerDied","Data":"de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223"}
Sep 19 08:54:25.745465 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.745451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r" event={"ID":"9bf6188c-b05a-442e-bca7-caf59eb22f7c","Type":"ContainerDied","Data":"2f54f0dfd0c0e5a5ae4d6135f10a8707e0b5193a54b4a027664e108d49631797"}
Sep 19 08:54:25.745586 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.745484    2580 scope.go:117] "RemoveContainer" containerID="de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223"
Sep 19 08:54:25.754641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.754622    2580 scope.go:117] "RemoveContainer" containerID="de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223"
Sep 19 08:54:25.754865 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:54:25.754846    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223\": container with ID starting with de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223 not found: ID does not exist" containerID="de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223"
Sep 19 08:54:25.754925 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.754874    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223"} err="failed to get container status \"de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223\": rpc error: code = NotFound desc = could not find container \"de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223\": container with ID starting with de1c1d974929ff64abbfc58a03b4511ae28c117e29164ebab8004ba00e170223 not found: ID does not exist"
Sep 19 08:54:25.767870 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.767842    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"]
Sep 19 08:54:25.771187 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:25.771164    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-9nb5r"]
Sep 19 08:54:26.079181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:26.079146    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" path="/var/lib/kubelet/pods/9bf6188c-b05a-442e-bca7-caf59eb22f7c/volumes"
Sep 19 08:54:30.512263 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512208    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"]
Sep 19 08:54:30.512707 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512671    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main"
Sep 19 08:54:30.512707 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512689    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main"
Sep 19 08:54:30.512827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512708    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="storage-initializer"
Sep 19 08:54:30.512827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512717    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="storage-initializer"
Sep 19 08:54:30.512827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512729    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main"
Sep 19 08:54:30.512827 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512738    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main"
Sep 19 08:54:30.513034 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512829    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bf6188c-b05a-442e-bca7-caf59eb22f7c" containerName="main"
Sep 19 08:54:30.513034 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.512846    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="47944175-0eb0-4181-924c-826a46737bb9" containerName="main"
Sep 19 08:54:30.516906 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.516885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.522140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.522118    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 19 08:54:30.522327 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.522306    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"router-gateway-1-openshift-default-dockercfg-8s2qv\""
Sep 19 08:54:30.522411 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.522344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 19 08:54:30.522411 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.522365    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-gw-ca-root-cert\""
Sep 19 08:54:30.546473 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.546440    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"]
Sep 19 08:54:30.589374 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589340    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589374 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589561 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589561 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589462    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589561 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589561 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589520    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589561 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpz2f\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-kube-api-access-cpz2f\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589721 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589609    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.589721 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.589635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8714fc-6be5-40de-a6eb-101992d2eff7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690229 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8714fc-6be5-40de-a6eb-101992d2eff7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690408 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690408 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690274    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690408 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690408 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690408 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690684 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690684 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpz2f\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-kube-api-access-cpz2f\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690684 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690832 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-certs\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690832 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690789    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-data\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.690909 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690838    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.691003 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.690974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8714fc-6be5-40de-a6eb-101992d2eff7-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.691121 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.691108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-credential-socket\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.692706 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.692686    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-envoy\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.692892 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.692873    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-podinfo\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.699313 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.699288    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-token\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.699419 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.699401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpz2f\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-kube-api-access-cpz2f\") pod \"router-gateway-1-openshift-default-6f9866b8dc-xtdxk\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") " pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:30.826392 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:30.826313    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:31.101131 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:31.101005    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"]
Sep 19 08:54:31.103458 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:54:31.103431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8714fc_6be5_40de_a6eb_101992d2eff7.slice/crio-4c3097442f2fc541cb91886c676c6e0aea68f95291b0609fe9cbb1f68d3b2e3a WatchSource:0}: Error finding container 4c3097442f2fc541cb91886c676c6e0aea68f95291b0609fe9cbb1f68d3b2e3a: Status 404 returned error can't find the container with id 4c3097442f2fc541cb91886c676c6e0aea68f95291b0609fe9cbb1f68d3b2e3a
Sep 19 08:54:31.105463 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:31.105441    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 08:54:31.776694 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:31.776649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" event={"ID":"9a8714fc-6be5-40de-a6eb-101992d2eff7","Type":"ContainerStarted","Data":"4c3097442f2fc541cb91886c676c6e0aea68f95291b0609fe9cbb1f68d3b2e3a"}
Sep 19 08:54:33.609960 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:33.609916    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 08:54:33.610298 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:33.610000    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 08:54:33.610298 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:33.610027    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 08:54:33.790905 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:33.790806    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" event={"ID":"9a8714fc-6be5-40de-a6eb-101992d2eff7","Type":"ContainerStarted","Data":"2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"}
Sep 19 08:54:33.810396 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:33.810335    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podStartSLOduration=1.306189553 podStartE2EDuration="3.810316585s" totalImagesPullingTime="2.504127032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:54:33.809779653 +0000 UTC m=+2025.995971610" watchObservedRunningTime="2026-09-19 08:54:33.810316585 +0000 UTC m=+2025.996508543"
Sep 19 08:54:33.826956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:33.826926    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:34.832187 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:34.832156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:35.801135 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:35.801097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:35.802135 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:35.802116    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:44.663835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.663793    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"]
Sep 19 08:54:44.668549 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.668523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.672094 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.672064    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 19 08:54:44.672094 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.672091    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-n7xsq\""
Sep 19 08:54:44.676355 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.676331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"]
Sep 19 08:54:44.705452 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.705420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.705595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.705474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.705595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.705537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.705595 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.705569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7cn\" (UniqueName: \"kubernetes.io/projected/59f62c47-fcad-4543-9938-d1a7e3e61dd9-kube-api-access-qh7cn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.705724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.705601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.705724 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.705618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806362 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806551 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7cn\" (UniqueName: \"kubernetes.io/projected/59f62c47-fcad-4543-9938-d1a7e3e61dd9-kube-api-access-qh7cn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806551 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806551 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806551 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806551 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806514    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.806933 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.807012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.807012 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.806975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.808769 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.808750    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.809147 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.809130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.813890 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.813872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7cn\" (UniqueName: \"kubernetes.io/projected/59f62c47-fcad-4543-9938-d1a7e3e61dd9-kube-api-access-qh7cn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:44.979384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:44.979352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:45.258888 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:45.258833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"]
Sep 19 08:54:45.261704 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:54:45.261673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f62c47_fcad_4543_9938_d1a7e3e61dd9.slice/crio-1e2c249751bbe173363c22499d550eab93232d9d59320dabb20cb37b62be3be1 WatchSource:0}: Error finding container 1e2c249751bbe173363c22499d550eab93232d9d59320dabb20cb37b62be3be1: Status 404 returned error can't find the container with id 1e2c249751bbe173363c22499d550eab93232d9d59320dabb20cb37b62be3be1
Sep 19 08:54:45.302973 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:54:45.302943    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f62c47_fcad_4543_9938_d1a7e3e61dd9.slice/crio-f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169.scope\": RecentStats: unable to find data in memory cache]"
Sep 19 08:54:45.844574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:45.844530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" event={"ID":"59f62c47-fcad-4543-9938-d1a7e3e61dd9","Type":"ContainerStarted","Data":"f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169"}
Sep 19 08:54:45.845025 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:45.844589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" event={"ID":"59f62c47-fcad-4543-9938-d1a7e3e61dd9","Type":"ContainerStarted","Data":"1e2c249751bbe173363c22499d550eab93232d9d59320dabb20cb37b62be3be1"}
Sep 19 08:54:45.861707 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:45.861655    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" podStartSLOduration=1.861639751 podStartE2EDuration="1.861639751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:54:45.859808668 +0000 UTC m=+2038.046000636" watchObservedRunningTime="2026-09-19 08:54:45.861639751 +0000 UTC m=+2038.047831765"
Sep 19 08:54:45.980163 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:45.980130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:54.979776 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:54.979727    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:54.992564 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:54.992536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:55.908396 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:55.908369    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:54:56.902833 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:56.902797    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerID="2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a" exitCode=0
Sep 19 08:54:56.903268 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:56.902876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" event={"ID":"9a8714fc-6be5-40de-a6eb-101992d2eff7","Type":"ContainerDied","Data":"2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"}
Sep 19 08:54:56.903352 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:56.903299    2580 scope.go:117] "RemoveContainer" containerID="2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"
Sep 19 08:54:56.904259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:56.904232    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 08:54:56.904340 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:56.904325    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 08:54:56.904384 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:56.904370    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 08:54:57.909914 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:57.909885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" event={"ID":"9a8714fc-6be5-40de-a6eb-101992d2eff7","Type":"ContainerStarted","Data":"7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511"}
Sep 19 08:54:58.827316 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:58.827272    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:54:58.827500 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:58.827468    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 19 08:54:59.827100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:54:59.827052    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 19 08:55:00.827210 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:00.827166    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 19 08:55:01.827300 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:01.827257    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 19 08:55:02.827215 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:02.827163    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 19 08:55:03.830880 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:03.830847    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:55:03.831280 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:03.831138    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:55:03.831705 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:03.831689    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:55:41.789502 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.789468    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"]
Sep 19 08:55:41.792817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.792783    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:41.796777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.796751    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 19 08:55:41.796920 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.796865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-krdsp\""
Sep 19 08:55:41.796920 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.796878    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 19 08:55:41.796920 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.796893    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 19 08:55:41.797074 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.796925    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-xrzv2\""
Sep 19 08:55:41.808311 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.808287    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"]
Sep 19 08:55:41.915214 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.915181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c1037-56d1-4280-815e-57e44752864c-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:41.915423 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:41.915268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8vm9\" (UniqueName: \"kubernetes.io/projected/e51c1037-56d1-4280-815e-57e44752864c-kube-api-access-j8vm9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:42.016237 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:42.016189    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c1037-56d1-4280-815e-57e44752864c-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:42.016402 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:42.016248    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8vm9\" (UniqueName: \"kubernetes.io/projected/e51c1037-56d1-4280-815e-57e44752864c-kube-api-access-j8vm9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:42.018744 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:42.018720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c1037-56d1-4280-815e-57e44752864c-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:42.026283 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:42.026263    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8vm9\" (UniqueName: \"kubernetes.io/projected/e51c1037-56d1-4280-815e-57e44752864c-kube-api-access-j8vm9\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:42.103393 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:42.103321    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:42.377124 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:42.377043    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"]
Sep 19 08:55:42.380777 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:55:42.380740    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51c1037_56d1_4280_815e_57e44752864c.slice/crio-3671404f38d695c522e04e17ef96a17901882a1c96d0b912aa7628e175ed800b WatchSource:0}: Error finding container 3671404f38d695c522e04e17ef96a17901882a1c96d0b912aa7628e175ed800b: Status 404 returned error can't find the container with id 3671404f38d695c522e04e17ef96a17901882a1c96d0b912aa7628e175ed800b
Sep 19 08:55:43.104409 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:43.104367    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" event={"ID":"e51c1037-56d1-4280-815e-57e44752864c","Type":"ContainerStarted","Data":"2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5"}
Sep 19 08:55:43.104787 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:43.104424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:55:43.104787 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:43.104447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" event={"ID":"e51c1037-56d1-4280-815e-57e44752864c","Type":"ContainerStarted","Data":"3671404f38d695c522e04e17ef96a17901882a1c96d0b912aa7628e175ed800b"}
Sep 19 08:55:43.123892 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:43.123843    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" podStartSLOduration=2.123826986 podStartE2EDuration="2.123826986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:55:43.122130754 +0000 UTC m=+2095.308322724" watchObservedRunningTime="2026-09-19 08:55:43.123826986 +0000 UTC m=+2095.310018978"
Sep 19 08:55:50.292089 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:55:50.292055    2580 scope.go:117] "RemoveContainer" containerID="9237889be44b850a00f8c920ccecfbf2bf2dd58e439bc02b042aef0a8b3fa11d"
Sep 19 08:56:13.979503 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:13.979466    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"]
Sep 19 08:56:13.988265 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:56:13.988229    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 08:56:13.988265 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:56:13.988265 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:56:13.988265 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169" cmd=["/bin/sleep","15"]
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:13.988288    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" message="" err=<
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:56:13.988306    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" podUID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" containerName="main" containerID="cri-o://f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169"
Sep 19 08:56:13.988553 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:13.988361    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" podUID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" containerName="main" containerID="cri-o://f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169" gracePeriod=60
Sep 19 08:56:14.110529 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.110508    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:56:14.239636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.239556    2580 generic.go:356] "Generic (PLEG): container finished" podID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" containerID="f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169" exitCode=0
Sep 19 08:56:14.239636 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.239620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" event={"ID":"59f62c47-fcad-4543-9938-d1a7e3e61dd9","Type":"ContainerDied","Data":"f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169"}
Sep 19 08:56:14.326277 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.326251    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:56:14.395140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395107    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tmp-dir\") pod \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") "
Sep 19 08:56:14.395140 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tls-certs\") pod \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") "
Sep 19 08:56:14.395442 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh7cn\" (UniqueName: \"kubernetes.io/projected/59f62c47-fcad-4543-9938-d1a7e3e61dd9-kube-api-access-qh7cn\") pod \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") "
Sep 19 08:56:14.395442 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395240    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-dshm\") pod \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") "
Sep 19 08:56:14.395442 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395256    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-home\") pod \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") "
Sep 19 08:56:14.395442 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-model-cache\") pod \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\" (UID: \"59f62c47-fcad-4543-9938-d1a7e3e61dd9\") "
Sep 19 08:56:14.395646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59f62c47-fcad-4543-9938-d1a7e3e61dd9" (UID: "59f62c47-fcad-4543-9938-d1a7e3e61dd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:14.395646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-home" (OuterVolumeSpecName: "home") pod "59f62c47-fcad-4543-9938-d1a7e3e61dd9" (UID: "59f62c47-fcad-4543-9938-d1a7e3e61dd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:14.395646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.395610    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "59f62c47-fcad-4543-9938-d1a7e3e61dd9" (UID: "59f62c47-fcad-4543-9938-d1a7e3e61dd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:14.397645 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.397614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-dshm" (OuterVolumeSpecName: "dshm") pod "59f62c47-fcad-4543-9938-d1a7e3e61dd9" (UID: "59f62c47-fcad-4543-9938-d1a7e3e61dd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:14.397645 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.397622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59f62c47-fcad-4543-9938-d1a7e3e61dd9" (UID: "59f62c47-fcad-4543-9938-d1a7e3e61dd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:56:14.397645 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.397633    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f62c47-fcad-4543-9938-d1a7e3e61dd9-kube-api-access-qh7cn" (OuterVolumeSpecName: "kube-api-access-qh7cn") pod "59f62c47-fcad-4543-9938-d1a7e3e61dd9" (UID: "59f62c47-fcad-4543-9938-d1a7e3e61dd9"). InnerVolumeSpecName "kube-api-access-qh7cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:56:14.496603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.496518    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7cn\" (UniqueName: \"kubernetes.io/projected/59f62c47-fcad-4543-9938-d1a7e3e61dd9-kube-api-access-qh7cn\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:14.496603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.496545    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:14.496603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.496554    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:14.496603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.496563    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:14.496603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.496572    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:14.496603 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:14.496579    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f62c47-fcad-4543-9938-d1a7e3e61dd9-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:15.245693 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:15.245654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx" event={"ID":"59f62c47-fcad-4543-9938-d1a7e3e61dd9","Type":"ContainerDied","Data":"1e2c249751bbe173363c22499d550eab93232d9d59320dabb20cb37b62be3be1"}
Sep 19 08:56:15.246123 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:15.245721    2580 scope.go:117] "RemoveContainer" containerID="f480a6aadfa203dabfcde088545d617c5044bd66fd865c7bd0196fac81faf169"
Sep 19 08:56:15.246123 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:15.245670    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"
Sep 19 08:56:15.266077 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:15.266047    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"]
Sep 19 08:56:15.270119 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:15.270092    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997nj2jx"]
Sep 19 08:56:16.080772 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:16.080739    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" path="/var/lib/kubelet/pods/59f62c47-fcad-4543-9938-d1a7e3e61dd9/volumes"
Sep 19 08:56:22.649633 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:22.649597    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"]
Sep 19 08:56:22.650090 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:22.649790    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy" containerID="cri-o://7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511" gracePeriod=30
Sep 19 08:56:24.224571 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.224537    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"]
Sep 19 08:56:24.224964 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.224891    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" containerName="main"
Sep 19 08:56:24.224964 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.224904    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" containerName="main"
Sep 19 08:56:24.225041 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.224998    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f62c47-fcad-4543-9938-d1a7e3e61dd9" containerName="main"
Sep 19 08:56:24.229293 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.229275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.232869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.232845    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-rmzdx\""
Sep 19 08:56:24.233001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.232962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 19 08:56:24.240253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.240203    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"]
Sep 19 08:56:24.279573 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.279540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-kube-api-access-tvlww\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.279752 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.279615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.380554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.380521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.380761 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.380592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-kube-api-access-tvlww\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.383292 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.383262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.400092 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.397105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-kube-api-access-tvlww\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.539707 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.539624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:24.811601 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:24.811360    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"]
Sep 19 08:56:24.813930 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:56:24.813896    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e82071_6c6c_41fa_b5eb_d0a1cf4f6714.slice/crio-d3fb08efe4f328bdcb2af7643a5af1ffa493495d6e36fc2f80a90769a98bee3c WatchSource:0}: Error finding container d3fb08efe4f328bdcb2af7643a5af1ffa493495d6e36fc2f80a90769a98bee3c: Status 404 returned error can't find the container with id d3fb08efe4f328bdcb2af7643a5af1ffa493495d6e36fc2f80a90769a98bee3c
Sep 19 08:56:25.292550 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:25.292505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" event={"ID":"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714","Type":"ContainerStarted","Data":"ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415"}
Sep 19 08:56:25.293009 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:25.292563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" event={"ID":"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714","Type":"ContainerStarted","Data":"d3fb08efe4f328bdcb2af7643a5af1ffa493495d6e36fc2f80a90769a98bee3c"}
Sep 19 08:56:25.293009 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:25.292592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:25.320705 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:25.320641    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" podStartSLOduration=1.320614312 podStartE2EDuration="1.320614312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:25.320450315 +0000 UTC m=+2137.506642273" watchObservedRunningTime="2026-09-19 08:56:25.320614312 +0000 UTC m=+2137.506806270"
Sep 19 08:56:27.895083 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:27.895055    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:56:28.011869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.011839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-certs\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.011889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-podinfo\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.011922    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpz2f\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-kube-api-access-cpz2f\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.011946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-token\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.011967    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8714fc-6be5-40de-a6eb-101992d2eff7-istiod-ca-cert\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012018 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012020    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-credential-socket\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-data\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012134    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:28.012338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012153    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-socket\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-envoy\") pod \"9a8714fc-6be5-40de-a6eb-101992d2eff7\" (UID: \"9a8714fc-6be5-40de-a6eb-101992d2eff7\") "
Sep 19 08:56:28.012554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:28.012554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:28.012554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012449    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8714fc-6be5-40de-a6eb-101992d2eff7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:56:28.012554 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012546    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.012765 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012564    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8714fc-6be5-40de-a6eb-101992d2eff7-istiod-ca-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.012765 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012576    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-credential-socket\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.012765 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012591    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-data\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.012765 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.012573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:28.014315 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.014287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:56:28.014443 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.014392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:56:28.014678 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.014656    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 08:56:28.014725 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.014662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-kube-api-access-cpz2f" (OuterVolumeSpecName: "kube-api-access-cpz2f") pod "9a8714fc-6be5-40de-a6eb-101992d2eff7" (UID: "9a8714fc-6be5-40de-a6eb-101992d2eff7"). InnerVolumeSpecName "kube-api-access-cpz2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:56:28.114028 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.114002    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-podinfo\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.114164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.114030    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpz2f\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-kube-api-access-cpz2f\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.114164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.114044    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-token\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.114164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.114058    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-workload-socket\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.114164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.114070    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8714fc-6be5-40de-a6eb-101992d2eff7-istio-envoy\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:56:28.307179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.307079    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerID="7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511" exitCode=0
Sep 19 08:56:28.307179 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.307130    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"
Sep 19 08:56:28.307457 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.307165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" event={"ID":"9a8714fc-6be5-40de-a6eb-101992d2eff7","Type":"ContainerDied","Data":"7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511"}
Sep 19 08:56:28.307457 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.307254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk" event={"ID":"9a8714fc-6be5-40de-a6eb-101992d2eff7","Type":"ContainerDied","Data":"4c3097442f2fc541cb91886c676c6e0aea68f95291b0609fe9cbb1f68d3b2e3a"}
Sep 19 08:56:28.307457 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.307285    2580 scope.go:117] "RemoveContainer" containerID="7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511"
Sep 19 08:56:28.315651 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.315634    2580 scope.go:117] "RemoveContainer" containerID="2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"
Sep 19 08:56:28.323404 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.323380    2580 scope.go:117] "RemoveContainer" containerID="7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511"
Sep 19 08:56:28.323691 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:56:28.323668    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511\": container with ID starting with 7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511 not found: ID does not exist" containerID="7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511"
Sep 19 08:56:28.323740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.323704    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511"} err="failed to get container status \"7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511\": rpc error: code = NotFound desc = could not find container \"7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511\": container with ID starting with 7fde1cb4c2229f347d19383c510cc99f0b8e23ed94baf66a73ed289a90e73511 not found: ID does not exist"
Sep 19 08:56:28.323740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.323726    2580 scope.go:117] "RemoveContainer" containerID="2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"
Sep 19 08:56:28.323829 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.323813    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"]
Sep 19 08:56:28.323965 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:56:28.323945    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a\": container with ID starting with 2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a not found: ID does not exist" containerID="2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"
Sep 19 08:56:28.324044 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.323972    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a"} err="failed to get container status \"2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a\": rpc error: code = NotFound desc = could not find container \"2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a\": container with ID starting with 2979c8522770bb870ec235832eb65767689ec031366ed7b39aee7fba22b4ca1a not found: ID does not exist"
Sep 19 08:56:28.329734 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:28.329713    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/router-gateway-1-openshift-default-6f9866b8dc-xtdxk"]
Sep 19 08:56:30.079751 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:30.079715    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" path="/var/lib/kubelet/pods/9a8714fc-6be5-40de-a6eb-101992d2eff7/volumes"
Sep 19 08:56:45.077293 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077256    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"]
Sep 19 08:56:45.077865 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077764    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy"
Sep 19 08:56:45.077865 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077784    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy"
Sep 19 08:56:45.077865 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077798    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy"
Sep 19 08:56:45.077865 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077808    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy"
Sep 19 08:56:45.078089 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077880    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy"
Sep 19 08:56:45.078089 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.077890    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a8714fc-6be5-40de-a6eb-101992d2eff7" containerName="istio-proxy"
Sep 19 08:56:45.080924 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.080901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.083550 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.083525    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 19 08:56:45.083688 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.083581    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-58jwz\""
Sep 19 08:56:45.083688 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.083535    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 19 08:56:45.083688 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.083536    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 08:56:45.083863 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.083687    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 19 08:56:45.083863 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.083792    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 19 08:56:45.090568 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.090542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"]
Sep 19 08:56:45.262574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262573    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262633    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262657    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.262835 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.262780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brpkl\" (UniqueName: \"kubernetes.io/projected/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kube-api-access-brpkl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364621 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364621 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364621 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364621 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brpkl\" (UniqueName: \"kubernetes.io/projected/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kube-api-access-brpkl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364817 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.364938 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.364829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.365113 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.365097    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.366673 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.366651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.366937 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.366919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.373044 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.373016    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brpkl\" (UniqueName: \"kubernetes.io/projected/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kube-api-access-brpkl\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.392164 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.392140    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:45.667492 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:45.667464    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"]
Sep 19 08:56:45.669974 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:56:45.669940    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fc7e2c_84fa_44c0_ac01_7ec4d0faac7b.slice/crio-7201097e11ec489ec3ec4aa41d4feee47f1ad099de32c69518b3af7544b13758 WatchSource:0}: Error finding container 7201097e11ec489ec3ec4aa41d4feee47f1ad099de32c69518b3af7544b13758: Status 404 returned error can't find the container with id 7201097e11ec489ec3ec4aa41d4feee47f1ad099de32c69518b3af7544b13758
Sep 19 08:56:46.389488 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:46.389453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" event={"ID":"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b","Type":"ContainerStarted","Data":"178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957"}
Sep 19 08:56:46.389948 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:46.389509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" event={"ID":"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b","Type":"ContainerStarted","Data":"7201097e11ec489ec3ec4aa41d4feee47f1ad099de32c69518b3af7544b13758"}
Sep 19 08:56:48.403253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:48.403138    2580 generic.go:356] "Generic (PLEG): container finished" podID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerID="178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957" exitCode=0
Sep 19 08:56:48.403253 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:48.403207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" event={"ID":"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b","Type":"ContainerDied","Data":"178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957"}
Sep 19 08:56:49.410019 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:49.409982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" event={"ID":"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b","Type":"ContainerStarted","Data":"76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da"}
Sep 19 08:56:49.434351 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:49.434290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podStartSLOduration=2.159554059 podStartE2EDuration="4.434270977s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.274716918s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:56:49.432620949 +0000 UTC m=+2161.618812905" watchObservedRunningTime="2026-09-19 08:56:49.434270977 +0000 UTC m=+2161.620462937"
Sep 19 08:56:50.392825 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:50.392784    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:55.393160 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:55.393119    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:56:55.394574 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:55.394542    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:56:56.298708 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:56.298679    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:56:59.613570 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:56:59.613532    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"]
Sep 19 08:57:05.393114 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:05.393007    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:57:06.298822 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:06.298787    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:57:14.635932 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:14.635881    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main" containerID="cri-o://ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415" gracePeriod=45
Sep 19 08:57:15.393533 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:15.393489    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:57:16.298956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:16.298913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:57:25.392641 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:25.392591    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:57:26.298567 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:26.298532    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:57:26.298740 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:26.298648    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:57:35.393379 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:35.393334    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:57:36.299658 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:36.299609    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:57:44.893401 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:44.893370    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:57:44.985408 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:44.985382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-kube-api-access-tvlww\") pod \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") "
Sep 19 08:57:44.985590 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:44.985424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-tls-certs\") pod \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\" (UID: \"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714\") "
Sep 19 08:57:44.987808 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:44.987779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-kube-api-access-tvlww" (OuterVolumeSpecName: "kube-api-access-tvlww") pod "63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" (UID: "63e82071-6c6c-41fa-b5eb-d0a1cf4f6714"). InnerVolumeSpecName "kube-api-access-tvlww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:57:44.988041 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:44.988016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" (UID: "63e82071-6c6c-41fa-b5eb-d0a1cf4f6714"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:57:45.086013 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.085980    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlww\" (UniqueName: \"kubernetes.io/projected/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-kube-api-access-tvlww\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:57:45.086013 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.086011    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:57:45.393323 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.393199    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:57:45.685078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.684984    2580 generic.go:356] "Generic (PLEG): container finished" podID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerID="ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415" exitCode=0
Sep 19 08:57:45.685078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.685033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" event={"ID":"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714","Type":"ContainerDied","Data":"ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415"}
Sep 19 08:57:45.685078 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.685038    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"
Sep 19 08:57:45.685375 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.685084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb" event={"ID":"63e82071-6c6c-41fa-b5eb-d0a1cf4f6714","Type":"ContainerDied","Data":"d3fb08efe4f328bdcb2af7643a5af1ffa493495d6e36fc2f80a90769a98bee3c"}
Sep 19 08:57:45.685375 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.685110    2580 scope.go:117] "RemoveContainer" containerID="ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415"
Sep 19 08:57:45.698127 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.698100    2580 scope.go:117] "RemoveContainer" containerID="ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415"
Sep 19 08:57:45.698467 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:57:45.698448    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415\": container with ID starting with ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415 not found: ID does not exist" containerID="ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415"
Sep 19 08:57:45.698535 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.698478    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415"} err="failed to get container status \"ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415\": rpc error: code = NotFound desc = could not find container \"ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415\": container with ID starting with ca0e97966614c283bb75eab7da88c6b554b75f5bfe14c45ff80cd3c345478415 not found: ID does not exist"
Sep 19 08:57:45.706133 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.706076    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"]
Sep 19 08:57:45.707869 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:45.707842    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche69ctb"]
Sep 19 08:57:46.079861 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:46.079819    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" path="/var/lib/kubelet/pods/63e82071-6c6c-41fa-b5eb-d0a1cf4f6714/volumes"
Sep 19 08:57:48.495777 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:48.495737    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"]
Sep 19 08:57:54.111125 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:54.111084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:57:55.393089 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:57:55.393033    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:58:03.515951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:03.515855    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main" containerID="cri-o://2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5" gracePeriod=45
Sep 19 08:58:04.111191 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:04.111154    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:58:05.393363 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:05.393311    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:58:14.111427 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:14.111385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:58:14.111908 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:14.111495    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:58:15.392772 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:15.392727    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:58:24.111199 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:24.111161    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 08:58:25.392812 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:25.392768    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:58:33.761324 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.761290    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:58:33.815593 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.815502    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c1037-56d1-4280-815e-57e44752864c-tls-certs\") pod \"e51c1037-56d1-4280-815e-57e44752864c\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") "
Sep 19 08:58:33.815758 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.815616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8vm9\" (UniqueName: \"kubernetes.io/projected/e51c1037-56d1-4280-815e-57e44752864c-kube-api-access-j8vm9\") pod \"e51c1037-56d1-4280-815e-57e44752864c\" (UID: \"e51c1037-56d1-4280-815e-57e44752864c\") "
Sep 19 08:58:33.817656 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.817631    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51c1037-56d1-4280-815e-57e44752864c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e51c1037-56d1-4280-815e-57e44752864c" (UID: "e51c1037-56d1-4280-815e-57e44752864c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:58:33.817826 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.817796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51c1037-56d1-4280-815e-57e44752864c-kube-api-access-j8vm9" (OuterVolumeSpecName: "kube-api-access-j8vm9") pod "e51c1037-56d1-4280-815e-57e44752864c" (UID: "e51c1037-56d1-4280-815e-57e44752864c"). InnerVolumeSpecName "kube-api-access-j8vm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:58:33.912021 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.911981    2580 generic.go:356] "Generic (PLEG): container finished" podID="e51c1037-56d1-4280-815e-57e44752864c" containerID="2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5" exitCode=0
Sep 19 08:58:33.912021 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.912026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"
Sep 19 08:58:33.912259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.912058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" event={"ID":"e51c1037-56d1-4280-815e-57e44752864c","Type":"ContainerDied","Data":"2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5"}
Sep 19 08:58:33.912259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.912124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq" event={"ID":"e51c1037-56d1-4280-815e-57e44752864c","Type":"ContainerDied","Data":"3671404f38d695c522e04e17ef96a17901882a1c96d0b912aa7628e175ed800b"}
Sep 19 08:58:33.912259 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.912149    2580 scope.go:117] "RemoveContainer" containerID="2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5"
Sep 19 08:58:33.916362 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.916340    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e51c1037-56d1-4280-815e-57e44752864c-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:58:33.916475 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.916363    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8vm9\" (UniqueName: \"kubernetes.io/projected/e51c1037-56d1-4280-815e-57e44752864c-kube-api-access-j8vm9\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:58:33.922193 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.922178    2580 scope.go:117] "RemoveContainer" containerID="2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5"
Sep 19 08:58:33.922578 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:58:33.922550    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5\": container with ID starting with 2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5 not found: ID does not exist" containerID="2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5"
Sep 19 08:58:33.922643 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.922593    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5"} err="failed to get container status \"2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5\": rpc error: code = NotFound desc = could not find container \"2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5\": container with ID starting with 2357ee2820df73c83a4bde6e403f6c2b99546b4307fd789e8e13a5e9d7a029e5 not found: ID does not exist"
Sep 19 08:58:33.937100 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.937067    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"]
Sep 19 08:58:33.940838 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:33.940808    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schejx2vq"]
Sep 19 08:58:34.079865 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:34.079786    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51c1037-56d1-4280-815e-57e44752864c" path="/var/lib/kubelet/pods/e51c1037-56d1-4280-815e-57e44752864c/volumes"
Sep 19 08:58:35.393270 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:35.393209    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 19 08:58:45.403241 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:45.403183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:58:45.411584 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:45.411556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:58:47.907820 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.907780    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"]
Sep 19 08:58:47.908330 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.908315    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main"
Sep 19 08:58:47.908421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.908335    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main"
Sep 19 08:58:47.908421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.908347    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main"
Sep 19 08:58:47.908421 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.908354    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main"
Sep 19 08:58:47.908579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.908462    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51c1037-56d1-4280-815e-57e44752864c" containerName="main"
Sep 19 08:58:47.908579 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.908477    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="63e82071-6c6c-41fa-b5eb-d0a1cf4f6714" containerName="main"
Sep 19 08:58:47.911405 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.911383    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:47.913951 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.913933    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 19 08:58:47.914780 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.914757    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-hwhm9\""
Sep 19 08:58:47.914954 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.914819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 19 08:58:47.914954 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.914842    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 19 08:58:47.914954 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.914919    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-26wqj\""
Sep 19 08:58:47.920137 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:47.920025    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"]
Sep 19 08:58:48.039001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.038958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdkv\" (UniqueName: \"kubernetes.io/projected/994194a5-0f36-4ced-8ace-a5cfdbd2392a-kube-api-access-8rdkv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.039001 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.039002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994194a5-0f36-4ced-8ace-a5cfdbd2392a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.139990 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.139960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdkv\" (UniqueName: \"kubernetes.io/projected/994194a5-0f36-4ced-8ace-a5cfdbd2392a-kube-api-access-8rdkv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.139990 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.139996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994194a5-0f36-4ced-8ace-a5cfdbd2392a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.142513 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.142490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994194a5-0f36-4ced-8ace-a5cfdbd2392a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.148734 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.148707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdkv\" (UniqueName: \"kubernetes.io/projected/994194a5-0f36-4ced-8ace-a5cfdbd2392a-kube-api-access-8rdkv\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.223544 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.223496    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.493906 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.493831    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"]
Sep 19 08:58:48.497711 ip-10-0-140-191 kubenswrapper[2580]: W0919 08:58:48.497679    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994194a5_0f36_4ced_8ace_a5cfdbd2392a.slice/crio-7f8d5f5d4809325e9378015f4dfd7d30688f3d7401b6b5ce9e85fbb739ba386c WatchSource:0}: Error finding container 7f8d5f5d4809325e9378015f4dfd7d30688f3d7401b6b5ce9e85fbb739ba386c: Status 404 returned error can't find the container with id 7f8d5f5d4809325e9378015f4dfd7d30688f3d7401b6b5ce9e85fbb739ba386c
Sep 19 08:58:48.978274 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.978235    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" event={"ID":"994194a5-0f36-4ced-8ace-a5cfdbd2392a","Type":"ContainerStarted","Data":"ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a"}
Sep 19 08:58:48.978702 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.978296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" event={"ID":"994194a5-0f36-4ced-8ace-a5cfdbd2392a","Type":"ContainerStarted","Data":"7f8d5f5d4809325e9378015f4dfd7d30688f3d7401b6b5ce9e85fbb739ba386c"}
Sep 19 08:58:48.978702 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.978326    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:58:48.994944 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:58:48.994894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podStartSLOduration=1.994880485 podStartE2EDuration="1.994880485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:58:48.993984294 +0000 UTC m=+2281.180176252" watchObservedRunningTime="2026-09-19 08:58:48.994880485 +0000 UTC m=+2281.181072449"
Sep 19 08:59:01.644043 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:01.644010    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"]
Sep 19 08:59:16.667841 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:16.667765    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main" containerID="cri-o://76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da" gracePeriod=45
Sep 19 08:59:17.336924 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.336897    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:59:17.403779 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403738    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-home\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.403956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-model-cache\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.403956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kserve-provision-location\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.403956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403848    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tmp-dir\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.403956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403874    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tls-certs\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.403956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403914    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brpkl\" (UniqueName: \"kubernetes.io/projected/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kube-api-access-brpkl\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.403956 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403940    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-cabundle-cert\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.404301 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.403968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-dshm\") pod \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\" (UID: \"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b\") "
Sep 19 08:59:17.404301 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.404145    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:17.404418 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.404339    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.404480 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.404407    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 08:59:17.404663 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.404622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-home" (OuterVolumeSpecName: "home") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:17.406472 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.406440    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kube-api-access-brpkl" (OuterVolumeSpecName: "kube-api-access-brpkl") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "kube-api-access-brpkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 08:59:17.406583 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.406537    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-dshm" (OuterVolumeSpecName: "dshm") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:17.406675 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.406565    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 08:59:17.422928 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.422856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:17.425952 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.425931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" (UID: "83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 08:59:17.505407 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505369    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brpkl\" (UniqueName: \"kubernetes.io/projected/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kube-api-access-brpkl\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.505407 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505403    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.505407 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505413    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.505626 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505421    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.505626 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505429    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.505626 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505453    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:17.505626 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:17.505462    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 08:59:18.104646 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.104608    2580 generic.go:356] "Generic (PLEG): container finished" podID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerID="76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da" exitCode=0
Sep 19 08:59:18.104999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.104659    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" event={"ID":"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b","Type":"ContainerDied","Data":"76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da"}
Sep 19 08:59:18.104999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.104668    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"
Sep 19 08:59:18.104999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.104711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh" event={"ID":"83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b","Type":"ContainerDied","Data":"7201097e11ec489ec3ec4aa41d4feee47f1ad099de32c69518b3af7544b13758"}
Sep 19 08:59:18.104999 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.104741    2580 scope.go:117] "RemoveContainer" containerID="76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da"
Sep 19 08:59:18.113181 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.113159    2580 scope.go:117] "RemoveContainer" containerID="178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957"
Sep 19 08:59:18.120879 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.120854    2580 scope.go:117] "RemoveContainer" containerID="76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da"
Sep 19 08:59:18.121175 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:59:18.121151    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da\": container with ID starting with 76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da not found: ID does not exist" containerID="76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da"
Sep 19 08:59:18.121269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.121183    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da"} err="failed to get container status \"76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da\": rpc error: code = NotFound desc = could not find container \"76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da\": container with ID starting with 76915db2801a07825e53a057f141cab170a216fd26b1635f80d68891404972da not found: ID does not exist"
Sep 19 08:59:18.121269 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.121204    2580 scope.go:117] "RemoveContainer" containerID="178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957"
Sep 19 08:59:18.121512 ip-10-0-140-191 kubenswrapper[2580]: E0919 08:59:18.121489    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957\": container with ID starting with 178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957 not found: ID does not exist" containerID="178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957"
Sep 19 08:59:18.121567 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.121518    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957"} err="failed to get container status \"178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957\": rpc error: code = NotFound desc = could not find container \"178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957\": container with ID starting with 178e47c96491944cc35fb6da302284187034e6d0b1da6555a767604f5d356957 not found: ID does not exist"
Sep 19 08:59:18.123060 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.123040    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"]
Sep 19 08:59:18.126494 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:18.126474    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94wfgwh"]
Sep 19 08:59:19.985255 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:19.985202    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 08:59:20.079540 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:20.079505    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" path="/var/lib/kubelet/pods/83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b/volumes"
Sep 19 08:59:59.670484 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.670442    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"]
Sep 19 08:59:59.672795 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.670920    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main"
Sep 19 08:59:59.672795 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.670937    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main"
Sep 19 08:59:59.672795 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.670955    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="storage-initializer"
Sep 19 08:59:59.672795 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.670964    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="storage-initializer"
Sep 19 08:59:59.672795 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.671064    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="83fc7e2c-84fa-44c0-ac01-7ec4d0faac7b" containerName="main"
Sep 19 08:59:59.674462 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.674445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.676767 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.676744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 19 08:59:59.676921 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.676901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 19 08:59:59.677002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.676914    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-gqndt\""
Sep 19 08:59:59.677002 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.676992    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 19 08:59:59.684830 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.684805    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"]
Sep 19 08:59:59.754790 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.754755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.754975 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.754807    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfmr\" (UniqueName: \"kubernetes.io/projected/c1822303-1ef0-4c80-9cbf-8307f0990374-kube-api-access-cwfmr\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.754975 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.754856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.754975 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.754890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.755165 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.754966    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.755165 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.755012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1822303-1ef0-4c80-9cbf-8307f0990374-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.755165 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.755061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.855887 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.855849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1822303-1ef0-4c80-9cbf-8307f0990374-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856098 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.855896    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856098 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.855952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856098 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfmr\" (UniqueName: \"kubernetes.io/projected/c1822303-1ef0-4c80-9cbf-8307f0990374-kube-api-access-cwfmr\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856098 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856098 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856433 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856491 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856491 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856568 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.856568 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.856554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.858260 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.858236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.858625 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.858605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1822303-1ef0-4c80-9cbf-8307f0990374-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.867624 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.867594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfmr\" (UniqueName: \"kubernetes.io/projected/c1822303-1ef0-4c80-9cbf-8307f0990374-kube-api-access-cwfmr\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 08:59:59.886338 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.886310    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"]
Sep 19 08:59:59.889849 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.889827    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 08:59:59.892174 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.892152    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-hb7hb\""
Sep 19 08:59:59.900277 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.900251    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"]
Sep 19 08:59:59.986644 ip-10-0-140-191 kubenswrapper[2580]: I0919 08:59:59.986617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:00:00.058096 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.058065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3858633e-de77-4de3-86a7-8c12e164e6a7-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.058266 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.058124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6kh\" (UniqueName: \"kubernetes.io/projected/3858633e-de77-4de3-86a7-8c12e164e6a7-kube-api-access-gz6kh\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.159259 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.159203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6kh\" (UniqueName: \"kubernetes.io/projected/3858633e-de77-4de3-86a7-8c12e164e6a7-kube-api-access-gz6kh\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.159463 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.159298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3858633e-de77-4de3-86a7-8c12e164e6a7-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.161799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.161774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3858633e-de77-4de3-86a7-8c12e164e6a7-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.167669 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.167646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6kh\" (UniqueName: \"kubernetes.io/projected/3858633e-de77-4de3-86a7-8c12e164e6a7-kube-api-access-gz6kh\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.201166 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.201126    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:00.262863 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.262670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"]
Sep 19 09:00:00.304131 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.304113    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 19 09:00:00.489739 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:00.489709    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"]
Sep 19 09:00:00.490679 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:00:00.490651    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3858633e_de77_4de3_86a7_8c12e164e6a7.slice/crio-1366b237715360ea10816bee8356a8891917ab7a21839f774504b518144f5c97 WatchSource:0}: Error finding container 1366b237715360ea10816bee8356a8891917ab7a21839f774504b518144f5c97: Status 404 returned error can't find the container with id 1366b237715360ea10816bee8356a8891917ab7a21839f774504b518144f5c97
Sep 19 09:00:01.286139 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:01.286105    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerStarted","Data":"94f005205b2dc80a57ff4b5317ac018db47a93b7536ff757536647255b585c88"}
Sep 19 09:00:01.287587 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:01.287563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" event={"ID":"3858633e-de77-4de3-86a7-8c12e164e6a7","Type":"ContainerStarted","Data":"e3c06e0331afd41c507a5cbfd1384f80df445ab5ce6869a4c6d408362cf3f840"}
Sep 19 09:00:01.287737 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:01.287601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" event={"ID":"3858633e-de77-4de3-86a7-8c12e164e6a7","Type":"ContainerStarted","Data":"1366b237715360ea10816bee8356a8891917ab7a21839f774504b518144f5c97"}
Sep 19 09:00:01.287737 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:01.287633    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:01.303952 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:01.303902    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" podStartSLOduration=2.303885243 podStartE2EDuration="2.303885243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:00:01.302854609 +0000 UTC m=+2353.489046568" watchObservedRunningTime="2026-09-19 09:00:01.303885243 +0000 UTC m=+2353.490077199"
Sep 19 09:00:06.313567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:06.313528    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerStarted","Data":"d747b938a0469a726fa54b801d77ec3f7e9122f7b8857f5cf83c0833b937b301"}
Sep 19 09:00:07.318907 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:07.318808    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerID="d747b938a0469a726fa54b801d77ec3f7e9122f7b8857f5cf83c0833b937b301" exitCode=0
Sep 19 09:00:07.319305 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:07.318897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerDied","Data":"d747b938a0469a726fa54b801d77ec3f7e9122f7b8857f5cf83c0833b937b301"}
Sep 19 09:00:08.325997 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:08.325902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerStarted","Data":"383b4043a46ef95a2e6419e6962aa10f3d7f15b3bf9c3caab6694d37af4c2a00"}
Sep 19 09:00:08.325997 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:08.325968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerStarted","Data":"4011e9f7299da2da169789b2232310b4f113e9a95ce66b314b1168142081900b"}
Sep 19 09:00:08.346697 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:08.346647    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podStartSLOduration=2.80092295 podStartE2EDuration="9.346627721s" totalImagesPullingTime="5.360961045s" totalInitContainerRuntime="1.184743726s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:59:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:00:08.345127185 +0000 UTC m=+2360.531319152" watchObservedRunningTime="2026-09-19 09:00:08.346627721 +0000 UTC m=+2360.532819678"
Sep 19 09:00:08.987587 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:08.987547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:00:09.987461 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:09.987424    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:00:09.988798 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:09.988768    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:00:19.987649 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:19.987609    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:00:29.987698 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:29.987654    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:00:32.294841 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:32.294812    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:00:34.946694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:34.946661    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"]
Sep 19 09:00:39.984989 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:39.984940    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:00:39.987392 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:39.987361    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:00:49.966153 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:49.966105    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main" containerID="cri-o://ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a" gracePeriod=45
Sep 19 09:00:49.984982 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:49.984941    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:00:49.987202 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:49.987181    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:00:59.984680 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:59.984640    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:00:59.985279 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:59.984747    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 09:00:59.987924 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:00:59.987902    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:01:09.984801 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:09.984721    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:01:09.987003 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:09.986973    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:01:19.984282 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:01:19.984250    2580 logging.go:55] [core] [Channel #788 SubChannel #789]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.59:9003", ServerName: "10.134.0.59:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.59:9003: connect: connection refused"
Sep 19 09:01:19.987673 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:19.987648    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:01:20.211608 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.211576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 09:01:20.256404 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.256321    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rdkv\" (UniqueName: \"kubernetes.io/projected/994194a5-0f36-4ced-8ace-a5cfdbd2392a-kube-api-access-8rdkv\") pod \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") "
Sep 19 09:01:20.256404 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.256358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994194a5-0f36-4ced-8ace-a5cfdbd2392a-tls-certs\") pod \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\" (UID: \"994194a5-0f36-4ced-8ace-a5cfdbd2392a\") "
Sep 19 09:01:20.258662 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.258632    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994194a5-0f36-4ced-8ace-a5cfdbd2392a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "994194a5-0f36-4ced-8ace-a5cfdbd2392a" (UID: "994194a5-0f36-4ced-8ace-a5cfdbd2392a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:01:20.258789 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.258658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994194a5-0f36-4ced-8ace-a5cfdbd2392a-kube-api-access-8rdkv" (OuterVolumeSpecName: "kube-api-access-8rdkv") pod "994194a5-0f36-4ced-8ace-a5cfdbd2392a" (UID: "994194a5-0f36-4ced-8ace-a5cfdbd2392a"). InnerVolumeSpecName "kube-api-access-8rdkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:01:20.357572 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.357542    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdkv\" (UniqueName: \"kubernetes.io/projected/994194a5-0f36-4ced-8ace-a5cfdbd2392a-kube-api-access-8rdkv\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:01:20.357572 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.357569    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/994194a5-0f36-4ced-8ace-a5cfdbd2392a-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:01:20.665125 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.665026    2580 generic.go:356] "Generic (PLEG): container finished" podID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerID="ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a" exitCode=0
Sep 19 09:01:20.665125 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.665073    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"
Sep 19 09:01:20.665125 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.665084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" event={"ID":"994194a5-0f36-4ced-8ace-a5cfdbd2392a","Type":"ContainerDied","Data":"ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a"}
Sep 19 09:01:20.665442 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.665143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" event={"ID":"994194a5-0f36-4ced-8ace-a5cfdbd2392a","Type":"ContainerDied","Data":"7f8d5f5d4809325e9378015f4dfd7d30688f3d7401b6b5ce9e85fbb739ba386c"}
Sep 19 09:01:20.665442 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.665174    2580 scope.go:117] "RemoveContainer" containerID="ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a"
Sep 19 09:01:20.675985 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.675958    2580 scope.go:117] "RemoveContainer" containerID="ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a"
Sep 19 09:01:20.676289 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:01:20.676266    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a\": container with ID starting with ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a not found: ID does not exist" containerID="ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a"
Sep 19 09:01:20.676376 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.676301    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a"} err="failed to get container status \"ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a\": rpc error: code = NotFound desc = could not find container \"ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a\": container with ID starting with ae53a096ce4fa48c5bddb00cc222c785b345074983ae721b68cd302e61420b2a not found: ID does not exist"
Sep 19 09:01:20.688442 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.688416    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"]
Sep 19 09:01:20.692081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.692049    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6"]
Sep 19 09:01:20.984338 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:20.984297    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schermfv6" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.59:9003\" within 1s: context deadline exceeded"
Sep 19 09:01:22.081059 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:22.081017    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" path="/var/lib/kubelet/pods/994194a5-0f36-4ced-8ace-a5cfdbd2392a/volumes"
Sep 19 09:01:29.987410 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:29.987359    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:01:37.666847 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.666808    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"]
Sep 19 09:01:37.667374 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.667331    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main"
Sep 19 09:01:37.667374 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.667348    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main"
Sep 19 09:01:37.667503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.667449    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="994194a5-0f36-4ced-8ace-a5cfdbd2392a" containerName="main"
Sep 19 09:01:37.670391 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.670366    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.672842 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.672811    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 19 09:01:37.673758 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.673735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-mm7r7\""
Sep 19 09:01:37.673927 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.673759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-8kxtf\""
Sep 19 09:01:37.674005 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.673769    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 19 09:01:37.674005 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.673831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 19 09:01:37.679548 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.679520    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"]
Sep 19 09:01:37.809004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.808962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05542e8d-237c-465f-8ca5-f7639af81acb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.809004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.809007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhsw\" (UniqueName: \"kubernetes.io/projected/05542e8d-237c-465f-8ca5-f7639af81acb-kube-api-access-mdhsw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.910268 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.910206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05542e8d-237c-465f-8ca5-f7639af81acb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.910268 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.910259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhsw\" (UniqueName: \"kubernetes.io/projected/05542e8d-237c-465f-8ca5-f7639af81acb-kube-api-access-mdhsw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.912761 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.912732    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05542e8d-237c-465f-8ca5-f7639af81acb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.919698 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.919625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhsw\" (UniqueName: \"kubernetes.io/projected/05542e8d-237c-465f-8ca5-f7639af81acb-kube-api-access-mdhsw\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:37.981682 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:37.981641    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:38.255912 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:38.255882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"]
Sep 19 09:01:38.258601 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:01:38.258574    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05542e8d_237c_465f_8ca5_f7639af81acb.slice/crio-3da3c920ab1316b5c91b4c8cc7dd5a987ec77d7a7e93f60772fdb6c79aea475f WatchSource:0}: Error finding container 3da3c920ab1316b5c91b4c8cc7dd5a987ec77d7a7e93f60772fdb6c79aea475f: Status 404 returned error can't find the container with id 3da3c920ab1316b5c91b4c8cc7dd5a987ec77d7a7e93f60772fdb6c79aea475f
Sep 19 09:01:38.758834 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:38.758794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" event={"ID":"05542e8d-237c-465f-8ca5-f7639af81acb","Type":"ContainerStarted","Data":"643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf"}
Sep 19 09:01:38.759352 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:38.758858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" event={"ID":"05542e8d-237c-465f-8ca5-f7639af81acb","Type":"ContainerStarted","Data":"3da3c920ab1316b5c91b4c8cc7dd5a987ec77d7a7e93f60772fdb6c79aea475f"}
Sep 19 09:01:38.759352 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:38.759011    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:01:38.776201 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:38.776148    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" podStartSLOduration=1.776131861 podStartE2EDuration="1.776131861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:01:38.774460739 +0000 UTC m=+2450.960652721" watchObservedRunningTime="2026-09-19 09:01:38.776131861 +0000 UTC m=+2450.962323821"
Sep 19 09:01:39.987102 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:39.987056    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:01:49.994081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:49.988114    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:01:59.997964 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:01:59.997933    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:02:00.005853 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:00.005824    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:02:04.895458 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:04.895420    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"]
Sep 19 09:02:04.901456 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:04.901424    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"]
Sep 19 09:02:04.901720 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:04.901693    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="modelcar" containerID="cri-o://383b4043a46ef95a2e6419e6962aa10f3d7f15b3bf9c3caab6694d37af4c2a00" gracePeriod=60
Sep 19 09:02:05.887495 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:05.887468    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd_c1822303-1ef0-4c80-9cbf-8307f0990374/modelcar/0.log"
Sep 19 09:02:05.888121 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:05.888100    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerID="383b4043a46ef95a2e6419e6962aa10f3d7f15b3bf9c3caab6694d37af4c2a00" exitCode=143
Sep 19 09:02:05.888199 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:05.888166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerDied","Data":"383b4043a46ef95a2e6419e6962aa10f3d7f15b3bf9c3caab6694d37af4c2a00"}
Sep 19 09:02:09.766266 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:09.766214    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:02:12.294306 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:12.294269    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:02:19.917231 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:19.917156    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main" containerID="cri-o://e3c06e0331afd41c507a5cbfd1384f80df445ab5ce6869a4c6d408362cf3f840" gracePeriod=45
Sep 19 09:02:19.926409 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:19.926376    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" containerID="cri-o://4011e9f7299da2da169789b2232310b4f113e9a95ce66b314b1168142081900b" gracePeriod=45
Sep 19 09:02:19.999271 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:19.999210    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Sep 19 09:02:20.956670 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:20.956639    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd_c1822303-1ef0-4c80-9cbf-8307f0990374/modelcar/0.log"
Sep 19 09:02:20.957456 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:20.957432    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerID="4011e9f7299da2da169789b2232310b4f113e9a95ce66b314b1168142081900b" exitCode=0
Sep 19 09:02:20.957541 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:20.957507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerDied","Data":"4011e9f7299da2da169789b2232310b4f113e9a95ce66b314b1168142081900b"}
Sep 19 09:02:21.179953 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.179932    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd_c1822303-1ef0-4c80-9cbf-8307f0990374/modelcar/0.log"
Sep 19 09:02:21.182721 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.182702    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:02:21.281336 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-tmp-dir\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281336 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1822303-1ef0-4c80-9cbf-8307f0990374-tls-certs\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281611 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281371    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-model-cache\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281611 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-dshm\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281611 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwfmr\" (UniqueName: \"kubernetes.io/projected/c1822303-1ef0-4c80-9cbf-8307f0990374-kube-api-access-cwfmr\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281781 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281623    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-home\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281781 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-kserve-provision-location\") pod \"c1822303-1ef0-4c80-9cbf-8307f0990374\" (UID: \"c1822303-1ef0-4c80-9cbf-8307f0990374\") "
Sep 19 09:02:21.281781 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:02:21.281953 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.281909    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.282081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.282048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:02:21.282345 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.282310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-home" (OuterVolumeSpecName: "home") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:02:21.283773 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.283745    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-dshm" (OuterVolumeSpecName: "dshm") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:02:21.283871 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.283749    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1822303-1ef0-4c80-9cbf-8307f0990374-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:02:21.283871 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.283771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1822303-1ef0-4c80-9cbf-8307f0990374-kube-api-access-cwfmr" (OuterVolumeSpecName: "kube-api-access-cwfmr") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "kube-api-access-cwfmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:02:21.292522 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.292500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1822303-1ef0-4c80-9cbf-8307f0990374" (UID: "c1822303-1ef0-4c80-9cbf-8307f0990374"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:02:21.383375 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.383326    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.383375 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.383371    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwfmr\" (UniqueName: \"kubernetes.io/projected/c1822303-1ef0-4c80-9cbf-8307f0990374-kube-api-access-cwfmr\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.383375 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.383381    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.383375 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.383390    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.383657 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.383401    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1822303-1ef0-4c80-9cbf-8307f0990374-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.383657 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.383410    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1822303-1ef0-4c80-9cbf-8307f0990374-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:21.963324 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.963300    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd_c1822303-1ef0-4c80-9cbf-8307f0990374/modelcar/0.log"
Sep 19 09:02:21.964035 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.964006    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" event={"ID":"c1822303-1ef0-4c80-9cbf-8307f0990374","Type":"ContainerDied","Data":"94f005205b2dc80a57ff4b5317ac018db47a93b7536ff757536647255b585c88"}
Sep 19 09:02:21.964155 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.964074    2580 scope.go:117] "RemoveContainer" containerID="383b4043a46ef95a2e6419e6962aa10f3d7f15b3bf9c3caab6694d37af4c2a00"
Sep 19 09:02:21.964155 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.964016    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"
Sep 19 09:02:21.972518 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.972497    2580 scope.go:117] "RemoveContainer" containerID="4011e9f7299da2da169789b2232310b4f113e9a95ce66b314b1168142081900b"
Sep 19 09:02:21.980068 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.980047    2580 scope.go:117] "RemoveContainer" containerID="d747b938a0469a726fa54b801d77ec3f7e9122f7b8857f5cf83c0833b937b301"
Sep 19 09:02:21.984631 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.984609    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"]
Sep 19 09:02:21.988994 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.988950    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd"]
Sep 19 09:02:21.999280 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:21.999250    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dgtvcd" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 19 09:02:22.083159 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:22.083125    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" path="/var/lib/kubelet/pods/c1822303-1ef0-4c80-9cbf-8307f0990374/volumes"
Sep 19 09:02:22.294986 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:22.294901    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:02:32.294829 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:32.294740    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:02:32.295212 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:32.294859    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:02:42.294734 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:42.294686    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:02:50.088086 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.088050    2580 generic.go:356] "Generic (PLEG): container finished" podID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerID="e3c06e0331afd41c507a5cbfd1384f80df445ab5ce6869a4c6d408362cf3f840" exitCode=0
Sep 19 09:02:50.088532 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.088114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" event={"ID":"3858633e-de77-4de3-86a7-8c12e164e6a7","Type":"ContainerDied","Data":"e3c06e0331afd41c507a5cbfd1384f80df445ab5ce6869a4c6d408362cf3f840"}
Sep 19 09:02:50.162583 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.162557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:02:50.219089 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.219064    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3858633e-de77-4de3-86a7-8c12e164e6a7-tls-certs\") pod \"3858633e-de77-4de3-86a7-8c12e164e6a7\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") "
Sep 19 09:02:50.219255 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.219121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz6kh\" (UniqueName: \"kubernetes.io/projected/3858633e-de77-4de3-86a7-8c12e164e6a7-kube-api-access-gz6kh\") pod \"3858633e-de77-4de3-86a7-8c12e164e6a7\" (UID: \"3858633e-de77-4de3-86a7-8c12e164e6a7\") "
Sep 19 09:02:50.221151 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.221123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3858633e-de77-4de3-86a7-8c12e164e6a7-kube-api-access-gz6kh" (OuterVolumeSpecName: "kube-api-access-gz6kh") pod "3858633e-de77-4de3-86a7-8c12e164e6a7" (UID: "3858633e-de77-4de3-86a7-8c12e164e6a7"). InnerVolumeSpecName "kube-api-access-gz6kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:02:50.221300 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.221279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3858633e-de77-4de3-86a7-8c12e164e6a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3858633e-de77-4de3-86a7-8c12e164e6a7" (UID: "3858633e-de77-4de3-86a7-8c12e164e6a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:02:50.320147 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.320118    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6kh\" (UniqueName: \"kubernetes.io/projected/3858633e-de77-4de3-86a7-8c12e164e6a7-kube-api-access-gz6kh\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:50.320147 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:50.320145    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3858633e-de77-4de3-86a7-8c12e164e6a7-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:02:51.094415 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:51.094378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm" event={"ID":"3858633e-de77-4de3-86a7-8c12e164e6a7","Type":"ContainerDied","Data":"1366b237715360ea10816bee8356a8891917ab7a21839f774504b518144f5c97"}
Sep 19 09:02:51.094887 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:51.094438    2580 scope.go:117] "RemoveContainer" containerID="e3c06e0331afd41c507a5cbfd1384f80df445ab5ce6869a4c6d408362cf3f840"
Sep 19 09:02:51.094887 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:51.094391    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"
Sep 19 09:02:51.116421 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:51.116394    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"]
Sep 19 09:02:51.119554 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:51.119531    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schemj5hm"]
Sep 19 09:02:52.079422 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:02:52.079385    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" path="/var/lib/kubelet/pods/3858633e-de77-4de3-86a7-8c12e164e6a7/volumes"
Sep 19 09:03:01.456765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.456731    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"]
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457043    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="modelcar"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457057    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="modelcar"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457067    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457072    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457081    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="modelcar-init"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457087    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="modelcar-init"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457103    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457108    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457164    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="main"
Sep 19 09:03:01.457167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457171    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3858633e-de77-4de3-86a7-8c12e164e6a7" containerName="main"
Sep 19 09:03:01.457550 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.457178    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1822303-1ef0-4c80-9cbf-8307f0990374" containerName="modelcar"
Sep 19 09:03:01.461595 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.461580    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.464926 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.464907    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:03:01.465062 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.464908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 19 09:03:01.465425 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.465403    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 19 09:03:01.465547 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.465466    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-c2qjs\""
Sep 19 09:03:01.465547 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.465407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 19 09:03:01.465681 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.465404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 19 09:03:01.474287 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.474265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"]
Sep 19 09:03:01.510890 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.510867    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-dshm\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511000 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.510905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5f8j\" (UniqueName: \"kubernetes.io/projected/3947f223-476d-415b-a69f-f33b555ff9af-kube-api-access-q5f8j\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511000 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.510934    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947f223-476d-415b-a69f-f33b555ff9af-tls-certs\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511000 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.510985    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-home\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.511012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3947f223-476d-415b-a69f-f33b555ff9af-cabundle-cert\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.511032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.511070    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-model-cache\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.511106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.511086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-tmp-dir\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612191 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947f223-476d-415b-a69f-f33b555ff9af-tls-certs\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612191 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-home\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3947f223-476d-415b-a69f-f33b555ff9af-cabundle-cert\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-model-cache\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-tmp-dir\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-dshm\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5f8j\" (UniqueName: \"kubernetes.io/projected/3947f223-476d-415b-a69f-f33b555ff9af-kube-api-access-q5f8j\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612707 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612653    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-home\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612792 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-tmp-dir\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612845 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-model-cache\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.612937 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.612907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.613052 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.613013    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3947f223-476d-415b-a69f-f33b555ff9af-cabundle-cert\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.614668 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.614646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-dshm\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.614990 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.614969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947f223-476d-415b-a69f-f33b555ff9af-tls-certs\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.622453 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.622434    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5f8j\" (UniqueName: \"kubernetes.io/projected/3947f223-476d-415b-a69f-f33b555ff9af-kube-api-access-q5f8j\") pod \"custom-route-timeout-test-kserve-667bcdf585-cqff2\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.661886 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.661851    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"]
Sep 19 09:03:01.666710 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.666681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.669141 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.669115    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-zv98l\""
Sep 19 09:03:01.673037 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.672981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"]
Sep 19 09:03:01.713880 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.713805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19008c9c-505a-4b8c-952f-87e4fbbaf116-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.713981 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.713892    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbhg\" (UniqueName: \"kubernetes.io/projected/19008c9c-505a-4b8c-952f-87e4fbbaf116-kube-api-access-wsbhg\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.771265 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.771212    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:01.814758 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.814721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19008c9c-505a-4b8c-952f-87e4fbbaf116-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.814932 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.814836    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbhg\" (UniqueName: \"kubernetes.io/projected/19008c9c-505a-4b8c-952f-87e4fbbaf116-kube-api-access-wsbhg\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.818510 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.817911    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19008c9c-505a-4b8c-952f-87e4fbbaf116-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.825153 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.825095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbhg\" (UniqueName: \"kubernetes.io/projected/19008c9c-505a-4b8c-952f-87e4fbbaf116-kube-api-access-wsbhg\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:01.977949 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:01.977852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:02.052196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:02.051441    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"]
Sep 19 09:03:02.054408 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:03:02.054374    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3947f223_476d_415b_a69f_f33b555ff9af.slice/crio-4f220af8ae7610653bb467a3d77ee564e47b91b41ffb6ab33f64deeb54964334 WatchSource:0}: Error finding container 4f220af8ae7610653bb467a3d77ee564e47b91b41ffb6ab33f64deeb54964334: Status 404 returned error can't find the container with id 4f220af8ae7610653bb467a3d77ee564e47b91b41ffb6ab33f64deeb54964334
Sep 19 09:03:02.141123 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:02.141090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" event={"ID":"3947f223-476d-415b-a69f-f33b555ff9af","Type":"ContainerStarted","Data":"b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb"}
Sep 19 09:03:02.141246 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:02.141145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" event={"ID":"3947f223-476d-415b-a69f-f33b555ff9af","Type":"ContainerStarted","Data":"4f220af8ae7610653bb467a3d77ee564e47b91b41ffb6ab33f64deeb54964334"}
Sep 19 09:03:02.247708 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:02.247620    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"]
Sep 19 09:03:02.252066 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:03:02.252037    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19008c9c_505a_4b8c_952f_87e4fbbaf116.slice/crio-cc6da2dff7f904be25e9fea6b68daab1ac3b5e586fb0e6dabdbd03266221a293 WatchSource:0}: Error finding container cc6da2dff7f904be25e9fea6b68daab1ac3b5e586fb0e6dabdbd03266221a293: Status 404 returned error can't find the container with id cc6da2dff7f904be25e9fea6b68daab1ac3b5e586fb0e6dabdbd03266221a293
Sep 19 09:03:03.148981 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:03.148934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" event={"ID":"19008c9c-505a-4b8c-952f-87e4fbbaf116","Type":"ContainerStarted","Data":"1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d"}
Sep 19 09:03:03.149526 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:03.149008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" event={"ID":"19008c9c-505a-4b8c-952f-87e4fbbaf116","Type":"ContainerStarted","Data":"cc6da2dff7f904be25e9fea6b68daab1ac3b5e586fb0e6dabdbd03266221a293"}
Sep 19 09:03:03.168830 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:03.168780    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" podStartSLOduration=2.168764589 podStartE2EDuration="2.168764589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:03:03.166466126 +0000 UTC m=+2535.352658085" watchObservedRunningTime="2026-09-19 09:03:03.168764589 +0000 UTC m=+2535.354956545"
Sep 19 09:03:04.155239 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:04.155187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:05.161088 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:05.161051    2580 generic.go:356] "Generic (PLEG): container finished" podID="3947f223-476d-415b-a69f-f33b555ff9af" containerID="b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb" exitCode=0
Sep 19 09:03:05.161522 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:05.161124    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" event={"ID":"3947f223-476d-415b-a69f-f33b555ff9af","Type":"ContainerDied","Data":"b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb"}
Sep 19 09:03:06.168433 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:06.168399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" event={"ID":"3947f223-476d-415b-a69f-f33b555ff9af","Type":"ContainerStarted","Data":"b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1"}
Sep 19 09:03:06.193267 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:06.193184    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podStartSLOduration=2.992141729 podStartE2EDuration="5.193162208s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.201020479s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:03:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:03:06.189660915 +0000 UTC m=+2538.375852872" watchObservedRunningTime="2026-09-19 09:03:06.193162208 +0000 UTC m=+2538.379354166"
Sep 19 09:03:06.772409 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:06.772370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:11.772442 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:11.772398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:03:11.774169 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:11.774141    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:03:21.772292 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:21.772211    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:03:22.392763 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:22.392720    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"]
Sep 19 09:03:29.766543 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:29.766503    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:03:31.772645 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:31.772598    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:03:35.163086 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:35.163056    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:03:37.413421 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:37.413357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main" containerID="cri-o://643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf" gracePeriod=45
Sep 19 09:03:39.766028 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:39.765980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:03:41.772356 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:41.772310    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:03:49.766133 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:49.766092    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:03:49.766560 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:49.766209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:03:51.771858 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:51.771812    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:03:59.766322 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:03:59.766281    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:04:01.771847 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:01.771795    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:04:07.660201 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.660173    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:04:07.787977 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.787937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdhsw\" (UniqueName: \"kubernetes.io/projected/05542e8d-237c-465f-8ca5-f7639af81acb-kube-api-access-mdhsw\") pod \"05542e8d-237c-465f-8ca5-f7639af81acb\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") "
Sep 19 09:04:07.788170 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.788012    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05542e8d-237c-465f-8ca5-f7639af81acb-tls-certs\") pod \"05542e8d-237c-465f-8ca5-f7639af81acb\" (UID: \"05542e8d-237c-465f-8ca5-f7639af81acb\") "
Sep 19 09:04:07.790249 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.790185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05542e8d-237c-465f-8ca5-f7639af81acb-kube-api-access-mdhsw" (OuterVolumeSpecName: "kube-api-access-mdhsw") pod "05542e8d-237c-465f-8ca5-f7639af81acb" (UID: "05542e8d-237c-465f-8ca5-f7639af81acb"). InnerVolumeSpecName "kube-api-access-mdhsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:04:07.790358 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.790243    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05542e8d-237c-465f-8ca5-f7639af81acb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05542e8d-237c-465f-8ca5-f7639af81acb" (UID: "05542e8d-237c-465f-8ca5-f7639af81acb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:04:07.888940 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.888901    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdhsw\" (UniqueName: \"kubernetes.io/projected/05542e8d-237c-465f-8ca5-f7639af81acb-kube-api-access-mdhsw\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:04:07.889106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:07.888960    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05542e8d-237c-465f-8ca5-f7639af81acb-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:04:08.448002 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.447970    2580 generic.go:356] "Generic (PLEG): container finished" podID="05542e8d-237c-465f-8ca5-f7639af81acb" containerID="643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf" exitCode=0
Sep 19 09:04:08.448169 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.448005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" event={"ID":"05542e8d-237c-465f-8ca5-f7639af81acb","Type":"ContainerDied","Data":"643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf"}
Sep 19 09:04:08.448169 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.448027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"
Sep 19 09:04:08.448169 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.448056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh" event={"ID":"05542e8d-237c-465f-8ca5-f7639af81acb","Type":"ContainerDied","Data":"3da3c920ab1316b5c91b4c8cc7dd5a987ec77d7a7e93f60772fdb6c79aea475f"}
Sep 19 09:04:08.448169 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.448081    2580 scope.go:117] "RemoveContainer" containerID="643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf"
Sep 19 09:04:08.456943 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.456921    2580 scope.go:117] "RemoveContainer" containerID="643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf"
Sep 19 09:04:08.457213 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:04:08.457193    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf\": container with ID starting with 643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf not found: ID does not exist" containerID="643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf"
Sep 19 09:04:08.457285 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.457233    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf"} err="failed to get container status \"643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf\": rpc error: code = NotFound desc = could not find container \"643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf\": container with ID starting with 643e96f6ada0799f5efb84b60195fa546ffd197670edfec015f277d07fc92acf not found: ID does not exist"
Sep 19 09:04:08.463922 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.463858    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"]
Sep 19 09:04:08.466557 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:08.466534    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejh6bh"]
Sep 19 09:04:10.080315 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:10.080283    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" path="/var/lib/kubelet/pods/05542e8d-237c-465f-8ca5-f7639af81acb/volumes"
Sep 19 09:04:11.772285 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:11.772216    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:04:21.772729 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:21.772684    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:04:31.771886 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:31.771821    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:04:41.772543 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:41.772496    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:04:51.772449 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:04:51.772401    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 19 09:05:01.781490 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:01.781460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:05:01.789399 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:01.789373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:05:06.910998 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:06.910962    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"]
Sep 19 09:05:06.915701 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:06.915671    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"]
Sep 19 09:05:15.163247 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:15.163188    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:05:21.940475 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:21.940430    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main" containerID="cri-o://b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1" gracePeriod=45
Sep 19 09:05:21.941943 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:21.941907    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main" containerID="cri-o://1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d" gracePeriod=45
Sep 19 09:05:22.685151 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.685125    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:05:22.796348 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.796248    2580 generic.go:356] "Generic (PLEG): container finished" podID="3947f223-476d-415b-a69f-f33b555ff9af" containerID="b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1" exitCode=0
Sep 19 09:05:22.796348 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.796293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" event={"ID":"3947f223-476d-415b-a69f-f33b555ff9af","Type":"ContainerDied","Data":"b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1"}
Sep 19 09:05:22.796348 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.796318    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"
Sep 19 09:05:22.796348 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.796334    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2" event={"ID":"3947f223-476d-415b-a69f-f33b555ff9af","Type":"ContainerDied","Data":"4f220af8ae7610653bb467a3d77ee564e47b91b41ffb6ab33f64deeb54964334"}
Sep 19 09:05:22.796684 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.796356    2580 scope.go:117] "RemoveContainer" containerID="b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1"
Sep 19 09:05:22.804637 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.804611    2580 scope.go:117] "RemoveContainer" containerID="b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb"
Sep 19 09:05:22.812059 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.812043    2580 scope.go:117] "RemoveContainer" containerID="b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1"
Sep 19 09:05:22.812363 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:05:22.812339    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1\": container with ID starting with b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1 not found: ID does not exist" containerID="b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1"
Sep 19 09:05:22.812449 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.812372    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1"} err="failed to get container status \"b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1\": rpc error: code = NotFound desc = could not find container \"b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1\": container with ID starting with b688ceed7be74e0d77fe74ad35d9629acb0798a577d5a72c1c382aeae767a6c1 not found: ID does not exist"
Sep 19 09:05:22.812449 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.812395    2580 scope.go:117] "RemoveContainer" containerID="b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb"
Sep 19 09:05:22.812681 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:05:22.812656    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb\": container with ID starting with b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb not found: ID does not exist" containerID="b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb"
Sep 19 09:05:22.812730 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.812688    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb"} err="failed to get container status \"b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb\": rpc error: code = NotFound desc = could not find container \"b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb\": container with ID starting with b9f33f67ec1ae9f473aa6243ec6a77b64a5eb47a8f1c60f43c864200e38b90fb not found: ID does not exist"
Sep 19 09:05:22.813872 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.813858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947f223-476d-415b-a69f-f33b555ff9af-tls-certs\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.813919 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.813889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-kserve-provision-location\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.813958 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.813930    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-home\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.813958 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.813949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-model-cache\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.814036 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.813976    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-dshm\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.814036 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3947f223-476d-415b-a69f-f33b555ff9af-cabundle-cert\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.814140 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814076    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-tmp-dir\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.814140 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814124    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5f8j\" (UniqueName: \"kubernetes.io/projected/3947f223-476d-415b-a69f-f33b555ff9af-kube-api-access-q5f8j\") pod \"3947f223-476d-415b-a69f-f33b555ff9af\" (UID: \"3947f223-476d-415b-a69f-f33b555ff9af\") "
Sep 19 09:05:22.814312 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814289    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-model-cache" (OuterVolumeSpecName: "model-cache") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:22.814467 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814438    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.814565 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3947f223-476d-415b-a69f-f33b555ff9af-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:05:22.814700 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.814674    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-home" (OuterVolumeSpecName: "home") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:22.816413 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.816385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3947f223-476d-415b-a69f-f33b555ff9af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:05:22.816671 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.816640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3947f223-476d-415b-a69f-f33b555ff9af-kube-api-access-q5f8j" (OuterVolumeSpecName: "kube-api-access-q5f8j") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "kube-api-access-q5f8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:05:22.816768 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.816733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-dshm" (OuterVolumeSpecName: "dshm") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:22.827891 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.827859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:22.838313 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.838288    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3947f223-476d-415b-a69f-f33b555ff9af" (UID: "3947f223-476d-415b-a69f-f33b555ff9af"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:05:22.915724 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915690    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3947f223-476d-415b-a69f-f33b555ff9af-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.915724 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915720    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.915915 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915733    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.915915 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915745    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.915915 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915756    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3947f223-476d-415b-a69f-f33b555ff9af-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.915915 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915768    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3947f223-476d-415b-a69f-f33b555ff9af-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:22.915915 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:22.915779    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5f8j\" (UniqueName: \"kubernetes.io/projected/3947f223-476d-415b-a69f-f33b555ff9af-kube-api-access-q5f8j\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:23.118803 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:23.118766    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"]
Sep 19 09:05:23.122848 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:23.122823    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-667bcdf585-cqff2"]
Sep 19 09:05:24.079373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:24.079342    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3947f223-476d-415b-a69f-f33b555ff9af" path="/var/lib/kubelet/pods/3947f223-476d-415b-a69f-f33b555ff9af/volumes"
Sep 19 09:05:25.163611 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:25.163563    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:05:35.163302 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:35.163193    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:05:35.163681 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:35.163330    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:05:45.162717 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:45.162680    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:05:52.191995 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.191968    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:05:52.261552 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.261470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsbhg\" (UniqueName: \"kubernetes.io/projected/19008c9c-505a-4b8c-952f-87e4fbbaf116-kube-api-access-wsbhg\") pod \"19008c9c-505a-4b8c-952f-87e4fbbaf116\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") "
Sep 19 09:05:52.261710 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.261568    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19008c9c-505a-4b8c-952f-87e4fbbaf116-tls-certs\") pod \"19008c9c-505a-4b8c-952f-87e4fbbaf116\" (UID: \"19008c9c-505a-4b8c-952f-87e4fbbaf116\") "
Sep 19 09:05:52.263544 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.263515    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19008c9c-505a-4b8c-952f-87e4fbbaf116-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19008c9c-505a-4b8c-952f-87e4fbbaf116" (UID: "19008c9c-505a-4b8c-952f-87e4fbbaf116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:05:52.263651 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.263596    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19008c9c-505a-4b8c-952f-87e4fbbaf116-kube-api-access-wsbhg" (OuterVolumeSpecName: "kube-api-access-wsbhg") pod "19008c9c-505a-4b8c-952f-87e4fbbaf116" (UID: "19008c9c-505a-4b8c-952f-87e4fbbaf116"). InnerVolumeSpecName "kube-api-access-wsbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:05:52.362835 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.362801    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19008c9c-505a-4b8c-952f-87e4fbbaf116-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:52.362835 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.362832    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbhg\" (UniqueName: \"kubernetes.io/projected/19008c9c-505a-4b8c-952f-87e4fbbaf116-kube-api-access-wsbhg\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:05:52.925470 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.925429    2580 generic.go:356] "Generic (PLEG): container finished" podID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerID="1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d" exitCode=0
Sep 19 09:05:52.925636 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.925480    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"
Sep 19 09:05:52.925636 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.925484    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" event={"ID":"19008c9c-505a-4b8c-952f-87e4fbbaf116","Type":"ContainerDied","Data":"1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d"}
Sep 19 09:05:52.925636 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.925536    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk" event={"ID":"19008c9c-505a-4b8c-952f-87e4fbbaf116","Type":"ContainerDied","Data":"cc6da2dff7f904be25e9fea6b68daab1ac3b5e586fb0e6dabdbd03266221a293"}
Sep 19 09:05:52.925636 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.925569    2580 scope.go:117] "RemoveContainer" containerID="1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d"
Sep 19 09:05:52.933970 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.933952    2580 scope.go:117] "RemoveContainer" containerID="1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d"
Sep 19 09:05:52.934272 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:05:52.934247    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d\": container with ID starting with 1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d not found: ID does not exist" containerID="1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d"
Sep 19 09:05:52.934362 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.934278    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d"} err="failed to get container status \"1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d\": rpc error: code = NotFound desc = could not find container \"1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d\": container with ID starting with 1d8b159908ebf3b9872fbf6f446f80f5ba5c7be9b14b2fbd921b465e33f3668d not found: ID does not exist"
Sep 19 09:05:52.947577 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.947549    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"]
Sep 19 09:05:52.950846 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:52.950822    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc58nh9vk"]
Sep 19 09:05:54.078901 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:54.078870    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" path="/var/lib/kubelet/pods/19008c9c-505a-4b8c-952f-87e4fbbaf116/volumes"
Sep 19 09:05:57.231837 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.231801    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"]
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232101    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232112    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232128    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232133    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232141    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="storage-initializer"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232146    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="storage-initializer"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232156    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main"
Sep 19 09:05:57.232187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232162    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main"
Sep 19 09:05:57.232492 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3947f223-476d-415b-a69f-f33b555ff9af" containerName="main"
Sep 19 09:05:57.232492 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232233    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="05542e8d-237c-465f-8ca5-f7639af81acb" containerName="main"
Sep 19 09:05:57.232492 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.232239    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="19008c9c-505a-4b8c-952f-87e4fbbaf116" containerName="main"
Sep 19 09:05:57.236480 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.236463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.239412 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.239392    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 19 09:05:57.239726 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.239711    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-gateway-1-openshift-default-dockercfg-hsvbf\""
Sep 19 09:05:57.240003 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.239986    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-gw-ca-root-cert\""
Sep 19 09:05:57.240133 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.240116    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 19 09:05:57.256352 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.256327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"]
Sep 19 09:05:57.301294 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301261    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301407 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301407 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301407 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301400    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301520 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301520 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301470    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301520 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301520 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.301644 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.301524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b54mg\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-kube-api-access-b54mg\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.402728 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.402881 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.402881 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.402881 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.402881 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.402881 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b54mg\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-kube-api-access-b54mg\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403159 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403159 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403159 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.402971    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403379 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.403245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-socket\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403379 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.403286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-credential-socket\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.403387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-certs\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.403405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-data\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.403620 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.403601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.405337 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.405319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-envoy\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.405660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.405639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-podinfo\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.411197 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.411177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-token\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.411614 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.411595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b54mg\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-kube-api-access-b54mg\") pod \"router-gateway-1-openshift-default-df55df88d-6h7jd\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") " pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.546523 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.546440    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:57.823056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.822972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"]
Sep 19 09:05:57.826300 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:05:57.826271    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408d88a4_bbc5_45cc_b05d_a90de701d3fc.slice/crio-8d15fb50caf3676f7ef4fefbb00c274c05289f02d5425aa7f8e88a9a80d37ba2 WatchSource:0}: Error finding container 8d15fb50caf3676f7ef4fefbb00c274c05289f02d5425aa7f8e88a9a80d37ba2: Status 404 returned error can't find the container with id 8d15fb50caf3676f7ef4fefbb00c274c05289f02d5425aa7f8e88a9a80d37ba2
Sep 19 09:05:57.828298 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.828265    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:05:57.828409 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.828341    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:05:57.828409 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.828384    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:05:57.947394 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.947357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" event={"ID":"408d88a4-bbc5-45cc-b05d-a90de701d3fc","Type":"ContainerStarted","Data":"bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"}
Sep 19 09:05:57.947587 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.947409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" event={"ID":"408d88a4-bbc5-45cc-b05d-a90de701d3fc","Type":"ContainerStarted","Data":"8d15fb50caf3676f7ef4fefbb00c274c05289f02d5425aa7f8e88a9a80d37ba2"}
Sep 19 09:05:57.966682 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:57.966634    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podStartSLOduration=0.966616916 podStartE2EDuration="966.616916ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:05:57.965772039 +0000 UTC m=+2710.151964006" watchObservedRunningTime="2026-09-19 09:05:57.966616916 +0000 UTC m=+2710.152808873"
Sep 19 09:05:58.546915 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:58.546877    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:58.551454 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:58.551430    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:58.951724 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:58.951692    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:05:58.952656 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:05:58.952631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:06:04.072624 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.072590    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"]
Sep 19 09:06:04.076329 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.076299    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.079127 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.079103    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 19 09:06:04.079270 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.079254    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:06:04.079341 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.079264    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-w4lzc\""
Sep 19 09:06:04.079341 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.079276    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 19 09:06:04.087875 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.087851    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"]
Sep 19 09:06:04.164298 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tls-certs\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-model-cache\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-home\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kserve-provision-location\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-dshm\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-cabundle-cert\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164489    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tmp-dir\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.164851 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.164547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgrd\" (UniqueName: \"kubernetes.io/projected/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kube-api-access-cvgrd\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265289 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kserve-provision-location\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265475 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265303    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-dshm\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265475 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-cabundle-cert\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265475 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265377    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tmp-dir\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265475 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgrd\" (UniqueName: \"kubernetes.io/projected/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kube-api-access-cvgrd\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265731 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tls-certs\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265731 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265531    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-model-cache\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.265731 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-home\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.266024 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.265997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-home\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.266329 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.266304    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kserve-provision-location\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.267243 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.267204    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tmp-dir\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.267413 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.267376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-cabundle-cert\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.267523 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.267508    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-model-cache\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.269724 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.269697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-dshm\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.269827 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.269728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tls-certs\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.274951 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.274929    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgrd\" (UniqueName: \"kubernetes.io/projected/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kube-api-access-cvgrd\") pod \"router-with-refs-test-kserve-5679bbc469-55ktf\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.349289 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.349147    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"]
Sep 19 09:06:04.361239 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.357662    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.364134 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.364099    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-hwzcl\""
Sep 19 09:06:04.370243 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.366279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"]
Sep 19 09:06:04.389373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.389350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:04.470564 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.470531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedd680a-4484-4870-a095-a08614f0b428-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.470893 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.470872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkf7v\" (UniqueName: \"kubernetes.io/projected/fedd680a-4484-4870-a095-a08614f0b428-kube-api-access-rkf7v\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.571798 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.571760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkf7v\" (UniqueName: \"kubernetes.io/projected/fedd680a-4484-4870-a095-a08614f0b428-kube-api-access-rkf7v\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.572076 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.571838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedd680a-4484-4870-a095-a08614f0b428-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.574545 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.574520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedd680a-4484-4870-a095-a08614f0b428-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.579507 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.579479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkf7v\" (UniqueName: \"kubernetes.io/projected/fedd680a-4484-4870-a095-a08614f0b428-kube-api-access-rkf7v\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.674041 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.674016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"]
Sep 19 09:06:04.676113 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:06:04.676084    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea59ac9_f50e_461e_b8af_e7a05f2cb1fc.slice/crio-d673ead2e810e9bdd9d95ee86e8505a102e82c23c71821e0490e202cc31dfc08 WatchSource:0}: Error finding container d673ead2e810e9bdd9d95ee86e8505a102e82c23c71821e0490e202cc31dfc08: Status 404 returned error can't find the container with id d673ead2e810e9bdd9d95ee86e8505a102e82c23c71821e0490e202cc31dfc08
Sep 19 09:06:04.679819 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.679796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:04.955720 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.955683    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"]
Sep 19 09:06:04.982372 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.982341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" event={"ID":"fedd680a-4484-4870-a095-a08614f0b428","Type":"ContainerStarted","Data":"a6ddde0266914709e787976da182dee02c996f935375cfed18ddd9059cd4eb93"}
Sep 19 09:06:04.984125 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.984098    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" event={"ID":"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc","Type":"ContainerStarted","Data":"67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809"}
Sep 19 09:06:04.984265 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:04.984145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" event={"ID":"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc","Type":"ContainerStarted","Data":"d673ead2e810e9bdd9d95ee86e8505a102e82c23c71821e0490e202cc31dfc08"}
Sep 19 09:06:05.990720 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:05.990671    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" event={"ID":"fedd680a-4484-4870-a095-a08614f0b428","Type":"ContainerStarted","Data":"165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766"}
Sep 19 09:06:05.991093 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:05.990767    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:06.007923 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:06.007878    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" podStartSLOduration=2.007864612 podStartE2EDuration="2.007864612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:06:06.006691469 +0000 UTC m=+2718.192883440" watchObservedRunningTime="2026-09-19 09:06:06.007864612 +0000 UTC m=+2718.194056569"
Sep 19 09:06:08.004363 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:08.004330    2580 generic.go:356] "Generic (PLEG): container finished" podID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerID="67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809" exitCode=0
Sep 19 09:06:08.004752 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:08.004414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" event={"ID":"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc","Type":"ContainerDied","Data":"67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809"}
Sep 19 09:06:09.010739 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.010664    2580 generic.go:356] "Generic (PLEG): container finished" podID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerID="bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a" exitCode=0
Sep 19 09:06:09.011191 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.010738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" event={"ID":"408d88a4-bbc5-45cc-b05d-a90de701d3fc","Type":"ContainerDied","Data":"bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"}
Sep 19 09:06:09.011288 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.011214    2580 scope.go:117] "RemoveContainer" containerID="bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"
Sep 19 09:06:09.012295 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.012257    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:06:09.012420 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.012330    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:06:09.012420 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.012364    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:06:09.012880 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.012860    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" event={"ID":"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc","Type":"ContainerStarted","Data":"f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2"}
Sep 19 09:06:09.064024 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.063960    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podStartSLOduration=2.628442399 podStartE2EDuration="5.063938098s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.435495699s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:06:09.062007464 +0000 UTC m=+2721.248199422" watchObservedRunningTime="2026-09-19 09:06:09.063938098 +0000 UTC m=+2721.250130056"
Sep 19 09:06:09.389607 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:09.389500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:10.019265 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:10.019208    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" event={"ID":"408d88a4-bbc5-45cc-b05d-a90de701d3fc","Type":"ContainerStarted","Data":"1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e"}
Sep 19 09:06:10.547035 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:10.546946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:06:10.548449 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:10.548409    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 19 09:06:11.547652 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:11.547594    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 19 09:06:12.546988 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:12.546925    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 19 09:06:13.547645 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:13.547599    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 19 09:06:13.952304 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:13.952267    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:06:14.390464 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:14.390371    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:06:14.391875 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:14.391847    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:06:14.546924 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:14.546878    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 19 09:06:15.551576 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:15.551547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:06:15.552613 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:15.552592    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:06:24.390625 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:24.390573    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:06:34.390658 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:34.390614    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:06:36.998381 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:36.998351    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:06:44.389802 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:44.389750    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:06:50.309499 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.309464    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"]
Sep 19 09:06:50.312682 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.312664    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.315484 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.315461    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 19 09:06:50.316450 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.316428    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 19 09:06:50.316586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.316454    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-ldc8n\""
Sep 19 09:06:50.316586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.316469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-85zwp\""
Sep 19 09:06:50.316586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.316519    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 19 09:06:50.321105 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.321083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"]
Sep 19 09:06:50.369410 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.369373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b84vt\" (UniqueName: \"kubernetes.io/projected/7a6af379-83e6-4a29-827a-cb422895428c-kube-api-access-b84vt\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.369410 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.369414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6af379-83e6-4a29-827a-cb422895428c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.469988 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.469954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6af379-83e6-4a29-827a-cb422895428c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.470115 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.470089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b84vt\" (UniqueName: \"kubernetes.io/projected/7a6af379-83e6-4a29-827a-cb422895428c-kube-api-access-b84vt\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.472400 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.472372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6af379-83e6-4a29-827a-cb422895428c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.478121 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.478099    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b84vt\" (UniqueName: \"kubernetes.io/projected/7a6af379-83e6-4a29-827a-cb422895428c-kube-api-access-b84vt\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.623999 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.623918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:50.901441 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:50.901414    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"]
Sep 19 09:06:50.904391 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:06:50.904357    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6af379_83e6_4a29_827a_cb422895428c.slice/crio-9f29c3a015d9e7b996429e8dcb84fb2b04ccbce863fa78fe399e6c755f39f30a WatchSource:0}: Error finding container 9f29c3a015d9e7b996429e8dcb84fb2b04ccbce863fa78fe399e6c755f39f30a: Status 404 returned error can't find the container with id 9f29c3a015d9e7b996429e8dcb84fb2b04ccbce863fa78fe399e6c755f39f30a
Sep 19 09:06:51.215567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:51.215532    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" event={"ID":"7a6af379-83e6-4a29-827a-cb422895428c","Type":"ContainerStarted","Data":"cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04"}
Sep 19 09:06:51.215736 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:51.215590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" event={"ID":"7a6af379-83e6-4a29-827a-cb422895428c","Type":"ContainerStarted","Data":"9f29c3a015d9e7b996429e8dcb84fb2b04ccbce863fa78fe399e6c755f39f30a"}
Sep 19 09:06:51.215784 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:51.215741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:06:51.232163 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:51.232114    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podStartSLOduration=1.232100097 podStartE2EDuration="1.232100097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:06:51.231176004 +0000 UTC m=+2763.417367960" watchObservedRunningTime="2026-09-19 09:06:51.232100097 +0000 UTC m=+2763.418292071"
Sep 19 09:06:54.390315 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:06:54.390260    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:07:04.390077 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:04.389984    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:07:14.390501 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:14.390456    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:07:22.224386 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:22.224354    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:07:24.390568 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:24.390524    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:07:34.390545 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:34.390487    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:07:44.389763 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:44.389708    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:07:54.390086 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:07:54.390032    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": dial tcp 10.134.0.66:8000: connect: connection refused"
Sep 19 09:08:04.400581 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:04.400547    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:08:04.408612 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:04.408591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:08:10.471783 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:10.471746    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"]
Sep 19 09:08:10.475998 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:10.475973    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"]
Sep 19 09:08:16.998666 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:16.998624    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:08:25.499504 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:25.499438    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main" containerID="cri-o://f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2" gracePeriod=45
Sep 19 09:08:25.500656 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:25.500629    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main" containerID="cri-o://165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766" gracePeriod=45
Sep 19 09:08:26.255168 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.255142    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:08:26.331296 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvgrd\" (UniqueName: \"kubernetes.io/projected/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kube-api-access-cvgrd\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331296 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331279    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kserve-provision-location\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331531 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tmp-dir\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331531 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-cabundle-cert\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331531 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331429    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-dshm\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331531 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-home\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331531 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331488    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-model-cache\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331531 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331521    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tls-certs\") pod \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\" (UID: \"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc\") "
Sep 19 09:08:26.331843 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.331781    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:08:26.332061 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.332026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:08:26.332388 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.332359    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-home" (OuterVolumeSpecName: "home") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:08:26.333964 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.333943    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:08:26.334097 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.334074    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kube-api-access-cvgrd" (OuterVolumeSpecName: "kube-api-access-cvgrd") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "kube-api-access-cvgrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:08:26.334146 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.334114    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-dshm" (OuterVolumeSpecName: "dshm") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:08:26.350460 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.350430    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:08:26.359675 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.359648    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" (UID: "cea59ac9-f50e-461e-b8af-e7a05f2cb1fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:08:26.432923 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432893    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.432923 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432919    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.432923 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432929    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.433137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432945    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgrd\" (UniqueName: \"kubernetes.io/projected/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kube-api-access-cvgrd\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.433137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432955    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.433137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432964    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.433137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432974    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.433137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.432981    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:26.667629 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.667540    2580 generic.go:356] "Generic (PLEG): container finished" podID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerID="f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2" exitCode=0
Sep 19 09:08:26.667629 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.667591    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"
Sep 19 09:08:26.668138 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.667629    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" event={"ID":"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc","Type":"ContainerDied","Data":"f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2"}
Sep 19 09:08:26.668138 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.667689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf" event={"ID":"cea59ac9-f50e-461e-b8af-e7a05f2cb1fc","Type":"ContainerDied","Data":"d673ead2e810e9bdd9d95ee86e8505a102e82c23c71821e0490e202cc31dfc08"}
Sep 19 09:08:26.668138 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.667720    2580 scope.go:117] "RemoveContainer" containerID="f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2"
Sep 19 09:08:26.677413 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.677395    2580 scope.go:117] "RemoveContainer" containerID="67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809"
Sep 19 09:08:26.684914 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.684897    2580 scope.go:117] "RemoveContainer" containerID="f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2"
Sep 19 09:08:26.685184 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:08:26.685165    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2\": container with ID starting with f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2 not found: ID does not exist" containerID="f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2"
Sep 19 09:08:26.685287 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.685193    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2"} err="failed to get container status \"f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2\": rpc error: code = NotFound desc = could not find container \"f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2\": container with ID starting with f45eada28134c62d23cab4fe38b9ed0491cd55aa37f03b60757c34dae3992ed2 not found: ID does not exist"
Sep 19 09:08:26.685287 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.685210    2580 scope.go:117] "RemoveContainer" containerID="67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809"
Sep 19 09:08:26.685500 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:08:26.685481    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809\": container with ID starting with 67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809 not found: ID does not exist" containerID="67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809"
Sep 19 09:08:26.685556 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.685506    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809"} err="failed to get container status \"67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809\": rpc error: code = NotFound desc = could not find container \"67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809\": container with ID starting with 67fddd21716aece6492b1056355a188dfeaa9596d1db36465f3f605b8add9809 not found: ID does not exist"
Sep 19 09:08:26.690137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.690114    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"]
Sep 19 09:08:26.693203 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.693179    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-55ktf"]
Sep 19 09:08:26.998371 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:26.998335    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:08:28.079444 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:28.079409    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" path="/var/lib/kubelet/pods/cea59ac9-f50e-461e-b8af-e7a05f2cb1fc/volumes"
Sep 19 09:08:36.998316 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:36.998203    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:08:36.998740 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:36.998342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:08:46.998283 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:46.998239    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:08:52.032784 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:52.032743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"]
Sep 19 09:08:52.223799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:52.223762    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:08:55.749452 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.749419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:08:55.776016 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.775951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedd680a-4484-4870-a095-a08614f0b428-tls-certs\") pod \"fedd680a-4484-4870-a095-a08614f0b428\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") "
Sep 19 09:08:55.776140 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.776050    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkf7v\" (UniqueName: \"kubernetes.io/projected/fedd680a-4484-4870-a095-a08614f0b428-kube-api-access-rkf7v\") pod \"fedd680a-4484-4870-a095-a08614f0b428\" (UID: \"fedd680a-4484-4870-a095-a08614f0b428\") "
Sep 19 09:08:55.778024 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.777996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedd680a-4484-4870-a095-a08614f0b428-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fedd680a-4484-4870-a095-a08614f0b428" (UID: "fedd680a-4484-4870-a095-a08614f0b428"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:08:55.778124 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.778027    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedd680a-4484-4870-a095-a08614f0b428-kube-api-access-rkf7v" (OuterVolumeSpecName: "kube-api-access-rkf7v") pod "fedd680a-4484-4870-a095-a08614f0b428" (UID: "fedd680a-4484-4870-a095-a08614f0b428"). InnerVolumeSpecName "kube-api-access-rkf7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:08:55.795164 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.795130    2580 generic.go:356] "Generic (PLEG): container finished" podID="fedd680a-4484-4870-a095-a08614f0b428" containerID="165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766" exitCode=0
Sep 19 09:08:55.795292 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.795172    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"
Sep 19 09:08:55.795292 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.795232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" event={"ID":"fedd680a-4484-4870-a095-a08614f0b428","Type":"ContainerDied","Data":"165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766"}
Sep 19 09:08:55.795428 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.795293    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd" event={"ID":"fedd680a-4484-4870-a095-a08614f0b428","Type":"ContainerDied","Data":"a6ddde0266914709e787976da182dee02c996f935375cfed18ddd9059cd4eb93"}
Sep 19 09:08:55.795428 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.795322    2580 scope.go:117] "RemoveContainer" containerID="165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766"
Sep 19 09:08:55.806247 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.806143    2580 scope.go:117] "RemoveContainer" containerID="165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766"
Sep 19 09:08:55.806803 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:08:55.806738    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766\": container with ID starting with 165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766 not found: ID does not exist" containerID="165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766"
Sep 19 09:08:55.806803 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.806769    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766"} err="failed to get container status \"165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766\": rpc error: code = NotFound desc = could not find container \"165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766\": container with ID starting with 165ebfdc9c6aec1c1fff1ab454b7b03ff0fb24a52525ef4435677dddd31c5766 not found: ID does not exist"
Sep 19 09:08:55.820694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.820671    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"]
Sep 19 09:08:55.829508 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.829487    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-snmgd"]
Sep 19 09:08:55.876895 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.876868    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkf7v\" (UniqueName: \"kubernetes.io/projected/fedd680a-4484-4870-a095-a08614f0b428-kube-api-access-rkf7v\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:55.876895 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:55.876891    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fedd680a-4484-4870-a095-a08614f0b428-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:08:56.080560 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:08:56.080480    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fedd680a-4484-4870-a095-a08614f0b428" path="/var/lib/kubelet/pods/fedd680a-4484-4870-a095-a08614f0b428/volumes"
Sep 19 09:09:02.224318 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:02.224278    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:09:05.499430 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:05.499387    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"]
Sep 19 09:09:05.499922 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:05.499658    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy" containerID="cri-o://1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e" gracePeriod=30
Sep 19 09:09:07.053537 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:07.053472    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main" containerID="cri-o://cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04" gracePeriod=45
Sep 19 09:09:10.753717 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.753689    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:09:10.864371 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.864335    2580 generic.go:356] "Generic (PLEG): container finished" podID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerID="1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e" exitCode=0
Sep 19 09:09:10.864539 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.864386    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"
Sep 19 09:09:10.864539 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.864414    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" event={"ID":"408d88a4-bbc5-45cc-b05d-a90de701d3fc","Type":"ContainerDied","Data":"1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e"}
Sep 19 09:09:10.864539 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.864465    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd" event={"ID":"408d88a4-bbc5-45cc-b05d-a90de701d3fc","Type":"ContainerDied","Data":"8d15fb50caf3676f7ef4fefbb00c274c05289f02d5425aa7f8e88a9a80d37ba2"}
Sep 19 09:09:10.864539 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.864491    2580 scope.go:117] "RemoveContainer" containerID="1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e"
Sep 19 09:09:10.872999 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.872983    2580 scope.go:117] "RemoveContainer" containerID="bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"
Sep 19 09:09:10.880372 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.880354    2580 scope.go:117] "RemoveContainer" containerID="1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e"
Sep 19 09:09:10.880628 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:09:10.880603    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e\": container with ID starting with 1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e not found: ID does not exist" containerID="1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e"
Sep 19 09:09:10.880733 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.880635    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e"} err="failed to get container status \"1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e\": rpc error: code = NotFound desc = could not find container \"1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e\": container with ID starting with 1f4fc887b33878c241b7d510b27f758c2d9049b9ee221b4ad215d1afd5752e8e not found: ID does not exist"
Sep 19 09:09:10.880733 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.880658    2580 scope.go:117] "RemoveContainer" containerID="bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"
Sep 19 09:09:10.880885 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:09:10.880869    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a\": container with ID starting with bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a not found: ID does not exist" containerID="bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"
Sep 19 09:09:10.880928 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.880889    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a"} err="failed to get container status \"bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a\": rpc error: code = NotFound desc = could not find container \"bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a\": container with ID starting with bc18608fc027d0d6fb5ec1c27f4dde6692716552ab2eba83dfe1fbed8cd5f41a not found: ID does not exist"
Sep 19 09:09:10.900341 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-certs\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900436 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900355    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-socket\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900436 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b54mg\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-kube-api-access-b54mg\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900436 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900431    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-credential-socket\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900560 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900493    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-token\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900560 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900553    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-data\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900583    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-envoy\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istiod-ca-cert\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900624    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:10.900660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900640    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-podinfo\") pod \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\" (UID: \"408d88a4-bbc5-45cc-b05d-a90de701d3fc\") "
Sep 19 09:09:10.900868 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900655    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:10.900868 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:10.900868 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900790    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:10.901056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900975    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-socket\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:10.901056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.900996    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-credential-socket\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:10.901056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.901011    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-data\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:10.901056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.901025    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-workload-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:10.901056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.901005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:09:10.902574 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.902546    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-kube-api-access-b54mg" (OuterVolumeSpecName: "kube-api-access-b54mg") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "kube-api-access-b54mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:09:10.902720 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.902699    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 09:09:10.902788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.902748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:09:10.902788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:10.902755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "408d88a4-bbc5-45cc-b05d-a90de701d3fc" (UID: "408d88a4-bbc5-45cc-b05d-a90de701d3fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:09:11.002373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.002328    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-envoy\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:11.002373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.002367    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istiod-ca-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:11.002373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.002377    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-podinfo\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:11.002373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.002385    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b54mg\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-kube-api-access-b54mg\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:11.002645 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.002395    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408d88a4-bbc5-45cc-b05d-a90de701d3fc-istio-token\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:11.187448 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.187416    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"]
Sep 19 09:09:11.192819 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:11.192796    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-gateway-1-openshift-default-df55df88d-6h7jd"]
Sep 19 09:09:12.079917 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:12.079881    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" path="/var/lib/kubelet/pods/408d88a4-bbc5-45cc-b05d-a90de701d3fc/volumes"
Sep 19 09:09:12.223660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:12.223612    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:09:12.223830 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:12.223716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:09:15.758089 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758057    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"]
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758372    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758384    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758396    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="storage-initializer"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758402    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="storage-initializer"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758410    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758417    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758429    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758435    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758442    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main"
Sep 19 09:09:15.758493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758447    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main"
Sep 19 09:09:15.758806 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758517    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedd680a-4484-4870-a095-a08614f0b428" containerName="main"
Sep 19 09:09:15.758806 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy"
Sep 19 09:09:15.758806 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758532    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="408d88a4-bbc5-45cc-b05d-a90de701d3fc" containerName="istio-proxy"
Sep 19 09:09:15.758806 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.758539    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cea59ac9-f50e-461e-b8af-e7a05f2cb1fc" containerName="main"
Sep 19 09:09:15.762997 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.762978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.766021 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.765998    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 19 09:09:15.766021 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.766006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 19 09:09:15.766234 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.766038    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:09:15.766332 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.766311    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 19 09:09:15.766884 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.766868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 19 09:09:15.766983 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.766871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-cpz7s\""
Sep 19 09:09:15.778213 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.778193    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"]
Sep 19 09:09:15.841887 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.841851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jkh\" (UniqueName: \"kubernetes.io/projected/caf3809b-a950-4721-9e53-74f531d2bb76-kube-api-access-m6jkh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.841887 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.841889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.842119 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.841921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.842119 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.842038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.842119 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.842080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3809b-a950-4721-9e53-74f531d2bb76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.842119 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.842117    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.842345 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.842135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caf3809b-a950-4721-9e53-74f531d2bb76-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.842345 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.842209    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.942969 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.942932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jkh\" (UniqueName: \"kubernetes.io/projected/caf3809b-a950-4721-9e53-74f531d2bb76-kube-api-access-m6jkh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.942969 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.942973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943252 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943252 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943054    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943252 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943081    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3809b-a950-4721-9e53-74f531d2bb76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943252 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943124    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943252 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943150    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caf3809b-a950-4721-9e53-74f531d2bb76-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943252 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943193    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.943917 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.943893    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caf3809b-a950-4721-9e53-74f531d2bb76-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.945571 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.945544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.945871 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.945849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3809b-a950-4721-9e53-74f531d2bb76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:15.950909 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:15.950889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jkh\" (UniqueName: \"kubernetes.io/projected/caf3809b-a950-4721-9e53-74f531d2bb76-kube-api-access-m6jkh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:16.073800 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:16.073714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:16.353893 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:16.353866    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"]
Sep 19 09:09:16.356001 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:09:16.355972    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf3809b_a950_4721_9e53_74f531d2bb76.slice/crio-9e6bc83addeb7548af21961fb0d6635ce8f1615d98ed5429ceb135deca7e2f77 WatchSource:0}: Error finding container 9e6bc83addeb7548af21961fb0d6635ce8f1615d98ed5429ceb135deca7e2f77: Status 404 returned error can't find the container with id 9e6bc83addeb7548af21961fb0d6635ce8f1615d98ed5429ceb135deca7e2f77
Sep 19 09:09:16.892444 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:16.892400    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" event={"ID":"caf3809b-a950-4721-9e53-74f531d2bb76","Type":"ContainerStarted","Data":"222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3"}
Sep 19 09:09:16.892891 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:16.892470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" event={"ID":"caf3809b-a950-4721-9e53-74f531d2bb76","Type":"ContainerStarted","Data":"9e6bc83addeb7548af21961fb0d6635ce8f1615d98ed5429ceb135deca7e2f77"}
Sep 19 09:09:18.902184 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:18.902145    2580 generic.go:356] "Generic (PLEG): container finished" podID="caf3809b-a950-4721-9e53-74f531d2bb76" containerID="222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3" exitCode=0
Sep 19 09:09:18.902610 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:18.902200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" event={"ID":"caf3809b-a950-4721-9e53-74f531d2bb76","Type":"ContainerDied","Data":"222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3"}
Sep 19 09:09:19.909447 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:19.909409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" event={"ID":"caf3809b-a950-4721-9e53-74f531d2bb76","Type":"ContainerStarted","Data":"5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d"}
Sep 19 09:09:19.944603 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:19.944547    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podStartSLOduration=2.665468751 podStartE2EDuration="4.944531228s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.279062477s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:09:19.944209539 +0000 UTC m=+2912.130401498" watchObservedRunningTime="2026-09-19 09:09:19.944531228 +0000 UTC m=+2912.130723193"
Sep 19 09:09:22.223456 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:22.223415    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:09:26.074284 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:26.074241    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:26.074669 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:26.074333    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:09:26.076145 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:26.076112    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:09:32.223912 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:32.223867    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:09:36.074204 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:36.074161    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:09:37.291911 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.291880    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:09:37.337524 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.337424    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b84vt\" (UniqueName: \"kubernetes.io/projected/7a6af379-83e6-4a29-827a-cb422895428c-kube-api-access-b84vt\") pod \"7a6af379-83e6-4a29-827a-cb422895428c\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") "
Sep 19 09:09:37.337524 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.337507    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6af379-83e6-4a29-827a-cb422895428c-tls-certs\") pod \"7a6af379-83e6-4a29-827a-cb422895428c\" (UID: \"7a6af379-83e6-4a29-827a-cb422895428c\") "
Sep 19 09:09:37.339687 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.339658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6af379-83e6-4a29-827a-cb422895428c-kube-api-access-b84vt" (OuterVolumeSpecName: "kube-api-access-b84vt") pod "7a6af379-83e6-4a29-827a-cb422895428c" (UID: "7a6af379-83e6-4a29-827a-cb422895428c"). InnerVolumeSpecName "kube-api-access-b84vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:09:37.339831 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.339680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6af379-83e6-4a29-827a-cb422895428c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a6af379-83e6-4a29-827a-cb422895428c" (UID: "7a6af379-83e6-4a29-827a-cb422895428c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:09:37.438600 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.438555    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a6af379-83e6-4a29-827a-cb422895428c-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:37.438600 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.438584    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b84vt\" (UniqueName: \"kubernetes.io/projected/7a6af379-83e6-4a29-827a-cb422895428c-kube-api-access-b84vt\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:09:37.993842 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.993802    2580 generic.go:356] "Generic (PLEG): container finished" podID="7a6af379-83e6-4a29-827a-cb422895428c" containerID="cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04" exitCode=0
Sep 19 09:09:37.994037 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.993850    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"
Sep 19 09:09:37.994037 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.993887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" event={"ID":"7a6af379-83e6-4a29-827a-cb422895428c","Type":"ContainerDied","Data":"cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04"}
Sep 19 09:09:37.994037 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.993939    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5" event={"ID":"7a6af379-83e6-4a29-827a-cb422895428c","Type":"ContainerDied","Data":"9f29c3a015d9e7b996429e8dcb84fb2b04ccbce863fa78fe399e6c755f39f30a"}
Sep 19 09:09:37.994037 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:37.993963    2580 scope.go:117] "RemoveContainer" containerID="cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04"
Sep 19 09:09:38.002884 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:38.002868    2580 scope.go:117] "RemoveContainer" containerID="cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04"
Sep 19 09:09:38.003172 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:09:38.003156    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04\": container with ID starting with cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04 not found: ID does not exist" containerID="cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04"
Sep 19 09:09:38.003237 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:38.003181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04"} err="failed to get container status \"cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04\": rpc error: code = NotFound desc = could not find container \"cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04\": container with ID starting with cdf118af2053ed7899f24cf9b109059fbe52ffbbde54f4037006b21a7c2f9c04 not found: ID does not exist"
Sep 19 09:09:38.015985 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:38.015956    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"]
Sep 19 09:09:38.021456 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:38.021436    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-t85p5"]
Sep 19 09:09:38.080760 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:38.080730    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6af379-83e6-4a29-827a-cb422895428c" path="/var/lib/kubelet/pods/7a6af379-83e6-4a29-827a-cb422895428c/volumes"
Sep 19 09:09:46.075012 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:46.074961    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:09:56.074784 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:09:56.074723    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:10:06.074514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:10:06.074412    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:10:16.074398 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:10:16.074341    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:10:26.074493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:10:26.074442    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:10:36.075254 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:10:36.075189    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:10:46.074164 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:10:46.074119    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:10:56.074861 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:10:56.074817    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:11:06.074765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:06.074712    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Sep 19 09:11:16.084904 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:16.084873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:11:16.092812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:16.092788    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:11:27.927992 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:27.927952    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"]
Sep 19 09:11:42.950678 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:42.950600    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main" containerID="cri-o://5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d" gracePeriod=45
Sep 19 09:11:43.693463 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.693437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:11:43.800478 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-model-cache\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800478 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800418    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-tmp-dir\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800478 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6jkh\" (UniqueName: \"kubernetes.io/projected/caf3809b-a950-4721-9e53-74f531d2bb76-kube-api-access-m6jkh\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-home\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-kserve-provision-location\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-dshm\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800585    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caf3809b-a950-4721-9e53-74f531d2bb76-cabundle-cert\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800638    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3809b-a950-4721-9e53-74f531d2bb76-tls-certs\") pod \"caf3809b-a950-4721-9e53-74f531d2bb76\" (UID: \"caf3809b-a950-4721-9e53-74f531d2bb76\") "
Sep 19 09:11:43.800765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-model-cache" (OuterVolumeSpecName: "model-cache") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:43.801097 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.800879    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.801285 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.801115    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-home" (OuterVolumeSpecName: "home") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:43.801449 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.801406    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf3809b-a950-4721-9e53-74f531d2bb76-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:11:43.803010 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.802979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf3809b-a950-4721-9e53-74f531d2bb76-kube-api-access-m6jkh" (OuterVolumeSpecName: "kube-api-access-m6jkh") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "kube-api-access-m6jkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:11:43.803128 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.803045    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-dshm" (OuterVolumeSpecName: "dshm") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:43.803196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.803118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf3809b-a950-4721-9e53-74f531d2bb76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:11:43.815061 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.815030    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:43.825008 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.824984    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "caf3809b-a950-4721-9e53-74f531d2bb76" (UID: "caf3809b-a950-4721-9e53-74f531d2bb76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:11:43.901304 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901273    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.901304 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901304    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6jkh\" (UniqueName: \"kubernetes.io/projected/caf3809b-a950-4721-9e53-74f531d2bb76-kube-api-access-m6jkh\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.901506 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901314    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.901506 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901323    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.901506 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901331    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf3809b-a950-4721-9e53-74f531d2bb76-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.901506 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901342    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caf3809b-a950-4721-9e53-74f531d2bb76-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:43.901506 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:43.901354    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf3809b-a950-4721-9e53-74f531d2bb76-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:11:44.547554 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.547516    2580 generic.go:356] "Generic (PLEG): container finished" podID="caf3809b-a950-4721-9e53-74f531d2bb76" containerID="5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d" exitCode=0
Sep 19 09:11:44.547554 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.547556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" event={"ID":"caf3809b-a950-4721-9e53-74f531d2bb76","Type":"ContainerDied","Data":"5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d"}
Sep 19 09:11:44.548094 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.547568    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"
Sep 19 09:11:44.548094 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.547598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm" event={"ID":"caf3809b-a950-4721-9e53-74f531d2bb76","Type":"ContainerDied","Data":"9e6bc83addeb7548af21961fb0d6635ce8f1615d98ed5429ceb135deca7e2f77"}
Sep 19 09:11:44.548094 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.547620    2580 scope.go:117] "RemoveContainer" containerID="5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d"
Sep 19 09:11:44.556108 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.556090    2580 scope.go:117] "RemoveContainer" containerID="222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3"
Sep 19 09:11:44.563339 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.563323    2580 scope.go:117] "RemoveContainer" containerID="5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d"
Sep 19 09:11:44.563584 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:11:44.563566    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d\": container with ID starting with 5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d not found: ID does not exist" containerID="5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d"
Sep 19 09:11:44.563632 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.563591    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d"} err="failed to get container status \"5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d\": rpc error: code = NotFound desc = could not find container \"5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d\": container with ID starting with 5c1d8d81dd2682b72d2590be7bcc58c46435489bbe7b43f578a617ceb8f6443d not found: ID does not exist"
Sep 19 09:11:44.563632 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.563607    2580 scope.go:117] "RemoveContainer" containerID="222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3"
Sep 19 09:11:44.563865 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:11:44.563844    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3\": container with ID starting with 222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3 not found: ID does not exist" containerID="222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3"
Sep 19 09:11:44.563908 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.563871    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3"} err="failed to get container status \"222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3\": rpc error: code = NotFound desc = could not find container \"222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3\": container with ID starting with 222553d0a40a5e47a0e0355cd11bbe77fe897bfd0adc1a75064c20962ff72ef3 not found: ID does not exist"
Sep 19 09:11:44.566926 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.566906    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"]
Sep 19 09:11:44.570455 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:44.570435    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f77cqsm"]
Sep 19 09:11:46.079407 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:11:46.079372    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" path="/var/lib/kubelet/pods/caf3809b-a950-4721-9e53-74f531d2bb76/volumes"
Sep 19 09:12:50.846968 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.846930    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"]
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847284    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847301    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847338    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="storage-initializer"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="storage-initializer"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847360    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847368    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847449    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf3809b-a950-4721-9e53-74f531d2bb76" containerName="main"
Sep 19 09:12:50.847514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.847468    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a6af379-83e6-4a29-827a-cb422895428c" containerName="main"
Sep 19 09:12:50.851797 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.851775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.855428 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.855404    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 19 09:12:50.855738 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.855713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-6kkx5\""
Sep 19 09:12:50.855879 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.855857    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-ml5kp\""
Sep 19 09:12:50.856184 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.856167    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:12:50.856342 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.856198    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 19 09:12:50.856444 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.856248    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 19 09:12:50.856651 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.856628    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 19 09:12:50.879999 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.879973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"]
Sep 19 09:12:50.931766 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-dshm\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.931916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.931916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.931916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931820    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.931916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrmb\" (UniqueName: \"kubernetes.io/projected/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kube-api-access-rxrmb\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.931916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.931916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-home\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:50.932118 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:50.931933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032623 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-home\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032627    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-dshm\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032707    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.032788 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrmb\" (UniqueName: \"kubernetes.io/projected/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kube-api-access-rxrmb\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.033106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.032810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.033106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.033074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-model-cache\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.033216 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.033129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.033216 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.033174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.033430 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.033406    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-home\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.033489 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.033431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.035688 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.035661    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-dshm\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.035881 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.035863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.043537 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.043520    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrmb\" (UniqueName: \"kubernetes.io/projected/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kube-api-access-rxrmb\") pod \"custom-route-timeout-pd-test-kserve-775694f4cd-s54g9\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.106166 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.106087    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"]
Sep 19 09:12:51.109494 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.109478    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.112325 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.112304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-5pmm8\""
Sep 19 09:12:51.124922 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.124896    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"]
Sep 19 09:12:51.164262 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.164213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.234944 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.234912    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b3359-b68d-41ec-8609-dbdf1be0c625-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.235088 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.234957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9vbr\" (UniqueName: \"kubernetes.io/projected/652b3359-b68d-41ec-8609-dbdf1be0c625-kube-api-access-r9vbr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.336046 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.336010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9vbr\" (UniqueName: \"kubernetes.io/projected/652b3359-b68d-41ec-8609-dbdf1be0c625-kube-api-access-r9vbr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.336215 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.336104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b3359-b68d-41ec-8609-dbdf1be0c625-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.338569 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.338549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b3359-b68d-41ec-8609-dbdf1be0c625-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.345496 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.345474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9vbr\" (UniqueName: \"kubernetes.io/projected/652b3359-b68d-41ec-8609-dbdf1be0c625-kube-api-access-r9vbr\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.420190 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.420125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.450711 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.450678    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"]
Sep 19 09:12:51.452039 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:12:51.452008    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20dfa19b_8804_48a3_b45f_3150f7b7fb4d.slice/crio-8e61b8ec73ee04733e54cb3a744e5160c0ed22161e3c87fabdc3bb079c12a4d4 WatchSource:0}: Error finding container 8e61b8ec73ee04733e54cb3a744e5160c0ed22161e3c87fabdc3bb079c12a4d4: Status 404 returned error can't find the container with id 8e61b8ec73ee04733e54cb3a744e5160c0ed22161e3c87fabdc3bb079c12a4d4
Sep 19 09:12:51.701699 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.701670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"]
Sep 19 09:12:51.703843 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:12:51.703817    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652b3359_b68d_41ec_8609_dbdf1be0c625.slice/crio-dfd5f9a6d5b58054a123383942cb4c96e3eb3e19ef01508f728792abe9a2eb07 WatchSource:0}: Error finding container dfd5f9a6d5b58054a123383942cb4c96e3eb3e19ef01508f728792abe9a2eb07: Status 404 returned error can't find the container with id dfd5f9a6d5b58054a123383942cb4c96e3eb3e19ef01508f728792abe9a2eb07
Sep 19 09:12:51.813536 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.813501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" event={"ID":"652b3359-b68d-41ec-8609-dbdf1be0c625","Type":"ContainerStarted","Data":"272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd"}
Sep 19 09:12:51.813737 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.813567    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:12:51.813737 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.813589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" event={"ID":"652b3359-b68d-41ec-8609-dbdf1be0c625","Type":"ContainerStarted","Data":"dfd5f9a6d5b58054a123383942cb4c96e3eb3e19ef01508f728792abe9a2eb07"}
Sep 19 09:12:51.815177 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.815148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerStarted","Data":"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"}
Sep 19 09:12:51.815302 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.815197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerStarted","Data":"8e61b8ec73ee04733e54cb3a744e5160c0ed22161e3c87fabdc3bb079c12a4d4"}
Sep 19 09:12:51.815302 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.815248    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:12:51.831976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:51.831920    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podStartSLOduration=0.831903589 podStartE2EDuration="831.903589ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:12:51.831440893 +0000 UTC m=+3124.017632852" watchObservedRunningTime="2026-09-19 09:12:51.831903589 +0000 UTC m=+3124.018095547"
Sep 19 09:12:52.827450 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:52.827398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerStarted","Data":"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"}
Sep 19 09:12:54.838377 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:54.838335    2580 generic.go:356] "Generic (PLEG): container finished" podID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerID="dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88" exitCode=0
Sep 19 09:12:54.838780 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:54.838409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerDied","Data":"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"}
Sep 19 09:12:55.845899 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:55.845864    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerStarted","Data":"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"}
Sep 19 09:12:55.874108 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:12:55.874057    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podStartSLOduration=3.721905487 podStartE2EDuration="5.874039571s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.152134084s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:12:55.871655135 +0000 UTC m=+3128.057847093" watchObservedRunningTime="2026-09-19 09:12:55.874039571 +0000 UTC m=+3128.060231527"
Sep 19 09:13:01.164526 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:01.164486    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:13:01.165092 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:01.164564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:13:01.165728 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:01.165689    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:13:01.176969 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:01.176940    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:13:11.165497 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:11.165307    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:13:21.164865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:21.164812    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:13:22.829900 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:22.829870    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:13:31.165567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:31.165518    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:13:41.164890 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:41.164835    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:13:51.165607 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:13:51.165544    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:14:01.165294 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:01.165248    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:14:11.165337 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:11.165284    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:14:21.165191 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:21.165137    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:14:31.165275 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:31.165201    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:14:41.165393 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:41.165344    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8001/health\": dial tcp 10.134.0.70:8001: connect: connection refused"
Sep 19 09:14:51.174538 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:51.174503    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:14:51.186737 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:14:51.186712    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:15:10.359393 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:10.359354    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"]
Sep 19 09:15:10.361010 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:10.360983    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"]
Sep 19 09:15:12.829320 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:12.829284    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:15:22.829833 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:22.829786    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:15:25.379748 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:25.379705    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main" containerID="cri-o://272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd" gracePeriod=45
Sep 19 09:15:25.381695 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:25.381643    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main" containerID="cri-o://16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45" gracePeriod=45
Sep 19 09:15:25.994670 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:25.994635    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="llm-d-routing-sidecar" containerID="cri-o://b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8" gracePeriod=45
Sep 19 09:15:26.235590 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.235560    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:15:26.409504 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409408    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-cabundle-cert\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.409504 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409453    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-home\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.409504 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409487    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tmp-dir\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxrmb\" (UniqueName: \"kubernetes.io/projected/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kube-api-access-rxrmb\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kserve-provision-location\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-model-cache\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409613    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-dshm\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tls-certs\") pod \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\" (UID: \"20dfa19b-8804-48a3-b45f-3150f7b7fb4d\") "
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.409852    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:15:26.410110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.410069    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.410545 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.410117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-model-cache" (OuterVolumeSpecName: "model-cache") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.410545 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.410311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-home" (OuterVolumeSpecName: "home") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.411983 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.411954    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kube-api-access-rxrmb" (OuterVolumeSpecName: "kube-api-access-rxrmb") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "kube-api-access-rxrmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:15:26.412500 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.412472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:15:26.412606 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.412495    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-dshm" (OuterVolumeSpecName: "dshm") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.422568 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.422532    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.433301 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.433281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20dfa19b-8804-48a3-b45f-3150f7b7fb4d" (UID: "20dfa19b-8804-48a3-b45f-3150f7b7fb4d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:15:26.510835 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510806    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.510835 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510830    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxrmb\" (UniqueName: \"kubernetes.io/projected/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kube-api-access-rxrmb\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.510835 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510840    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.511032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510848    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.511032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510857    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.511032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510866    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.511032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.510875    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20dfa19b-8804-48a3-b45f-3150f7b7fb4d-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:26.536612 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536580    2580 generic.go:356] "Generic (PLEG): container finished" podID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerID="16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45" exitCode=0
Sep 19 09:15:26.536612 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536605    2580 generic.go:356] "Generic (PLEG): container finished" podID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerID="b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8" exitCode=0
Sep 19 09:15:26.536807 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536648    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"
Sep 19 09:15:26.536807 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerDied","Data":"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"}
Sep 19 09:15:26.536807 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerDied","Data":"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"}
Sep 19 09:15:26.536807 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9" event={"ID":"20dfa19b-8804-48a3-b45f-3150f7b7fb4d","Type":"ContainerDied","Data":"8e61b8ec73ee04733e54cb3a744e5160c0ed22161e3c87fabdc3bb079c12a4d4"}
Sep 19 09:15:26.536807 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.536756    2580 scope.go:117] "RemoveContainer" containerID="16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"
Sep 19 09:15:26.545378 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.545351    2580 scope.go:117] "RemoveContainer" containerID="dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"
Sep 19 09:15:26.553163 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.553143    2580 scope.go:117] "RemoveContainer" containerID="b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"
Sep 19 09:15:26.559479 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.559455    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"]
Sep 19 09:15:26.560778 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.560763    2580 scope.go:117] "RemoveContainer" containerID="16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"
Sep 19 09:15:26.561049 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:15:26.561030    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45\": container with ID starting with 16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45 not found: ID does not exist" containerID="16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"
Sep 19 09:15:26.561143 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561062    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"} err="failed to get container status \"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45\": rpc error: code = NotFound desc = could not find container \"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45\": container with ID starting with 16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45 not found: ID does not exist"
Sep 19 09:15:26.561143 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561101    2580 scope.go:117] "RemoveContainer" containerID="dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"
Sep 19 09:15:26.561376 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:15:26.561356    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88\": container with ID starting with dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88 not found: ID does not exist" containerID="dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"
Sep 19 09:15:26.561429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561383    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"} err="failed to get container status \"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88\": rpc error: code = NotFound desc = could not find container \"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88\": container with ID starting with dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88 not found: ID does not exist"
Sep 19 09:15:26.561429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561398    2580 scope.go:117] "RemoveContainer" containerID="b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"
Sep 19 09:15:26.561648 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:15:26.561629    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8\": container with ID starting with b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8 not found: ID does not exist" containerID="b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"
Sep 19 09:15:26.561719 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561657    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"} err="failed to get container status \"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8\": rpc error: code = NotFound desc = could not find container \"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8\": container with ID starting with b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8 not found: ID does not exist"
Sep 19 09:15:26.561719 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561677    2580 scope.go:117] "RemoveContainer" containerID="16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"
Sep 19 09:15:26.561913 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561892    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45"} err="failed to get container status \"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45\": rpc error: code = NotFound desc = could not find container \"16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45\": container with ID starting with 16ff1291a5bb17fb1bbeb60401708b683dd08e04b8b2712a405dbf2553251f45 not found: ID does not exist"
Sep 19 09:15:26.561963 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.561914    2580 scope.go:117] "RemoveContainer" containerID="dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"
Sep 19 09:15:26.562172 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.562150    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88"} err="failed to get container status \"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88\": rpc error: code = NotFound desc = could not find container \"dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88\": container with ID starting with dcbfa92aa4b020127ff8e83cc69329978313fc8ec84459e36d789d749faf1d88 not found: ID does not exist"
Sep 19 09:15:26.562267 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.562172    2580 scope.go:117] "RemoveContainer" containerID="b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"
Sep 19 09:15:26.562545 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.562507    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8"} err="failed to get container status \"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8\": rpc error: code = NotFound desc = could not find container \"b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8\": container with ID starting with b902218fc2af1235aeec439da9bfa84c87d6a24b49e695c2d7f0381ec6fb10e8 not found: ID does not exist"
Sep 19 09:15:26.564170 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:26.564134    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-775694f4cd-s54g9"]
Sep 19 09:15:28.079617 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:28.079582    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" path="/var/lib/kubelet/pods/20dfa19b-8804-48a3-b45f-3150f7b7fb4d/volumes"
Sep 19 09:15:32.829238 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:32.829180    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:15:32.829625 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:32.829312    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:15:42.829297 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:42.829255    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:15:52.829099 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:52.829055    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:15:55.620288 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.620264    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:15:55.662585 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.662501    2580 generic.go:356] "Generic (PLEG): container finished" podID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerID="272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd" exitCode=0
Sep 19 09:15:55.662585 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.662557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"
Sep 19 09:15:55.662750 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.662562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" event={"ID":"652b3359-b68d-41ec-8609-dbdf1be0c625","Type":"ContainerDied","Data":"272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd"}
Sep 19 09:15:55.662750 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.662697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k" event={"ID":"652b3359-b68d-41ec-8609-dbdf1be0c625","Type":"ContainerDied","Data":"dfd5f9a6d5b58054a123383942cb4c96e3eb3e19ef01508f728792abe9a2eb07"}
Sep 19 09:15:55.662750 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.662722    2580 scope.go:117] "RemoveContainer" containerID="272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd"
Sep 19 09:15:55.670741 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.670726    2580 scope.go:117] "RemoveContainer" containerID="272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd"
Sep 19 09:15:55.671001 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:15:55.670974    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd\": container with ID starting with 272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd not found: ID does not exist" containerID="272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd"
Sep 19 09:15:55.671066 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.671001    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd"} err="failed to get container status \"272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd\": rpc error: code = NotFound desc = could not find container \"272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd\": container with ID starting with 272b0cb6d51b318b035c637304be8e4a99b8478e1bdc8e433f0cbadc9f57e3cd not found: ID does not exist"
Sep 19 09:15:55.751888 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.751867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9vbr\" (UniqueName: \"kubernetes.io/projected/652b3359-b68d-41ec-8609-dbdf1be0c625-kube-api-access-r9vbr\") pod \"652b3359-b68d-41ec-8609-dbdf1be0c625\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") "
Sep 19 09:15:55.752021 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.751901    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b3359-b68d-41ec-8609-dbdf1be0c625-tls-certs\") pod \"652b3359-b68d-41ec-8609-dbdf1be0c625\" (UID: \"652b3359-b68d-41ec-8609-dbdf1be0c625\") "
Sep 19 09:15:55.753943 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.753918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652b3359-b68d-41ec-8609-dbdf1be0c625-kube-api-access-r9vbr" (OuterVolumeSpecName: "kube-api-access-r9vbr") pod "652b3359-b68d-41ec-8609-dbdf1be0c625" (UID: "652b3359-b68d-41ec-8609-dbdf1be0c625"). InnerVolumeSpecName "kube-api-access-r9vbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:15:55.754057 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.754036    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652b3359-b68d-41ec-8609-dbdf1be0c625-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "652b3359-b68d-41ec-8609-dbdf1be0c625" (UID: "652b3359-b68d-41ec-8609-dbdf1be0c625"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:15:55.853109 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.853073    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9vbr\" (UniqueName: \"kubernetes.io/projected/652b3359-b68d-41ec-8609-dbdf1be0c625-kube-api-access-r9vbr\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:55.853109 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.853104    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/652b3359-b68d-41ec-8609-dbdf1be0c625-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:15:55.984738 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.984706    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"]
Sep 19 09:15:55.989357 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:55.989335    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c485sr9k"]
Sep 19 09:15:56.079712 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.079684    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" path="/var/lib/kubelet/pods/652b3359-b68d-41ec-8609-dbdf1be0c625/volumes"
Sep 19 09:15:56.376138 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376049    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"]
Sep 19 09:15:56.376385 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376370    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="llm-d-routing-sidecar"
Sep 19 09:15:56.376385 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376385    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="llm-d-routing-sidecar"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376401    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376407    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376416    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="storage-initializer"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376422    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="storage-initializer"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376430    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376435    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376491    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="main"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376498    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="20dfa19b-8804-48a3-b45f-3150f7b7fb4d" containerName="llm-d-routing-sidecar"
Sep 19 09:15:56.376516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.376505    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="652b3359-b68d-41ec-8609-dbdf1be0c625" containerName="main"
Sep 19 09:15:56.380699 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.380675    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.382870 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.382848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-6dx5h\""
Sep 19 09:15:56.382981 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.382940    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 19 09:15:56.383504 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.383489    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 19 09:15:56.383604 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.383490    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 19 09:15:56.397801 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.397778    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"]
Sep 19 09:15:56.458180 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458180 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458361 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458202    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458361 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458361 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnspx\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-kube-api-access-bnspx\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458464 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458464 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458464 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.458464 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.458430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559422 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559493 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559578 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559560    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559626 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559613    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnspx\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-kube-api-access-bnspx\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559671 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559660    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559710 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559710 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559703    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559817 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559736    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.559817 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.559800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.560061 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.560034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.560138 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.560096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.560182 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.560130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.560380 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.560362    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.561820 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.561794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.562013 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.561997    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.571697 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.571668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.571808 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.571790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnspx\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-kube-api-access-bnspx\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-fgcjr\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.691006 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.690919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:56.980372 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.980338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"]
Sep 19 09:15:56.983265 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:15:56.983232    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3ae3d1_1da5_4aaa_9b72_ba999cec60da.slice/crio-e28cb3812d0633c84fc9315706436c93b9ea6b39426ff971baa77f3dd9e3b4a0 WatchSource:0}: Error finding container e28cb3812d0633c84fc9315706436c93b9ea6b39426ff971baa77f3dd9e3b4a0: Status 404 returned error can't find the container with id e28cb3812d0633c84fc9315706436c93b9ea6b39426ff971baa77f3dd9e3b4a0
Sep 19 09:15:56.985212 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.985182    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:15:56.985307 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.985264    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:15:56.985307 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:56.985296    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:15:57.672768 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:57.672733    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" event={"ID":"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da","Type":"ContainerStarted","Data":"cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"}
Sep 19 09:15:57.672989 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:57.672783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" event={"ID":"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da","Type":"ContainerStarted","Data":"e28cb3812d0633c84fc9315706436c93b9ea6b39426ff971baa77f3dd9e3b4a0"}
Sep 19 09:15:57.690995 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:57.690967    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:57.692490 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:57.692448    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" podStartSLOduration=1.692432449 podStartE2EDuration="1.692432449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:15:57.691275077 +0000 UTC m=+3309.877467036" watchObservedRunningTime="2026-09-19 09:15:57.692432449 +0000 UTC m=+3309.878624406"
Sep 19 09:15:58.696901 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:58.696872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:59.682505 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:59.682416    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:15:59.743249 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:15:59.743181    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:16:08.724560 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:08.724521    2580 generic.go:356] "Generic (PLEG): container finished" podID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerID="cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e" exitCode=0
Sep 19 09:16:08.725062 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:08.724586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" event={"ID":"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da","Type":"ContainerDied","Data":"cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"}
Sep 19 09:16:08.725062 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:08.724928    2580 scope.go:117] "RemoveContainer" containerID="cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"
Sep 19 09:16:08.725946 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:08.725912    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:16:08.726011 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:08.725996    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:16:08.726050 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:08.726039    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 19 09:16:09.731385 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:09.731350    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" event={"ID":"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da","Type":"ContainerStarted","Data":"7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d"}
Sep 19 09:16:10.691230 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:10.691193    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:16:10.692403 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:10.692369    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 09:16:11.692301 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:11.692259    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 09:16:12.691801 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:12.691760    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 09:16:13.417979 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.417939    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"]
Sep 19 09:16:13.421453 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.421433    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.424319 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.424293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 19 09:16:13.424935 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.424734    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-pjczb\""
Sep 19 09:16:13.424935 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.424746    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 19 09:16:13.424935 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.424799    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:16:13.424935 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.424831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-l9x84\""
Sep 19 09:16:13.440541 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.440505    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"]
Sep 19 09:16:13.504474 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504474 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-home\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504505    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4966ab-6e00-4b00-863e-ccb449203c88-tls-certs\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504571    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9sg\" (UniqueName: \"kubernetes.io/projected/cd4966ab-6e00-4b00-863e-ccb449203c88-kube-api-access-7v9sg\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd4966ab-6e00-4b00-863e-ccb449203c88-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-model-cache\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-dshm\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.504694 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.504644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.605764 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605714    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd4966ab-6e00-4b00-863e-ccb449203c88-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.605764 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605766    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-model-cache\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606047 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-dshm\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606047 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606047 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605855    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606047 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-home\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606047 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.605947    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4966ab-6e00-4b00-863e-ccb449203c88-tls-certs\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606360 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.606118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9sg\" (UniqueName: \"kubernetes.io/projected/cd4966ab-6e00-4b00-863e-ccb449203c88-kube-api-access-7v9sg\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606421 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.606395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606489 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.606422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-model-cache\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606548 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.606493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-home\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606548 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.606505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.606736 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.606716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd4966ab-6e00-4b00-863e-ccb449203c88-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.608250 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.608209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-dshm\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.608571 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.608555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4966ab-6e00-4b00-863e-ccb449203c88-tls-certs\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.619690 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.619664    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9sg\" (UniqueName: \"kubernetes.io/projected/cd4966ab-6e00-4b00-863e-ccb449203c88-kube-api-access-7v9sg\") pod \"router-with-refs-pd-test-kserve-6757b868d4-sjv92\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.691757 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.691719    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 19 09:16:13.736973 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.736938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:13.958297 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.953839    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"]
Sep 19 09:16:13.968705 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.968591    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"]
Sep 19 09:16:13.968705 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.968697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:13.971257 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:13.971213    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-lspk5\""
Sep 19 09:16:14.029462 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.029388    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"]
Sep 19 09:16:14.031733 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:16:14.031706    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4966ab_6e00_4b00_863e_ccb449203c88.slice/crio-5ea22635fb385088fad0e8b1bd29557f75428dc4388552c3bc2ea63f442e1683 WatchSource:0}: Error finding container 5ea22635fb385088fad0e8b1bd29557f75428dc4388552c3bc2ea63f442e1683: Status 404 returned error can't find the container with id 5ea22635fb385088fad0e8b1bd29557f75428dc4388552c3bc2ea63f442e1683
Sep 19 09:16:14.110027 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.110001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8nk\" (UniqueName: \"kubernetes.io/projected/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-kube-api-access-nh8nk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.110169 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.110144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.211411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.211329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.211411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.211392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8nk\" (UniqueName: \"kubernetes.io/projected/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-kube-api-access-nh8nk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.213914 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.213885    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.220183 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.220159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8nk\" (UniqueName: \"kubernetes.io/projected/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-kube-api-access-nh8nk\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.280246 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.280184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.552497 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.552395    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"]
Sep 19 09:16:14.557483 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:16:14.557451    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f302ea_f9e0_4fd0_b6fa_ff678dd1db4f.slice/crio-72169e72c005dd8765117a57e557869581f941eca3d2604fae092eab9f14b7ee WatchSource:0}: Error finding container 72169e72c005dd8765117a57e557869581f941eca3d2604fae092eab9f14b7ee: Status 404 returned error can't find the container with id 72169e72c005dd8765117a57e557869581f941eca3d2604fae092eab9f14b7ee
Sep 19 09:16:14.682983 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.682949    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:16:14.756539 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.756499    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" event={"ID":"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f","Type":"ContainerStarted","Data":"883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051"}
Sep 19 09:16:14.756752 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.756561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" event={"ID":"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f","Type":"ContainerStarted","Data":"72169e72c005dd8765117a57e557869581f941eca3d2604fae092eab9f14b7ee"}
Sep 19 09:16:14.756752 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.756631    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:14.758014 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.757991    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerStarted","Data":"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"}
Sep 19 09:16:14.758149 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.758026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerStarted","Data":"5ea22635fb385088fad0e8b1bd29557f75428dc4388552c3bc2ea63f442e1683"}
Sep 19 09:16:14.758149 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.758130    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:14.774142 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.774106    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podStartSLOduration=1.774092379 podStartE2EDuration="1.774092379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:16:14.772361096 +0000 UTC m=+3326.958553087" watchObservedRunningTime="2026-09-19 09:16:14.774092379 +0000 UTC m=+3326.960284342"
Sep 19 09:16:14.815589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.815556    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:16:14.816981 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:14.816959    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:16:15.767309 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:15.767271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerStarted","Data":"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"}
Sep 19 09:16:17.780667 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:17.780631    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerID="95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd" exitCode=0
Sep 19 09:16:17.781108 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:17.780703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerDied","Data":"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"}
Sep 19 09:16:18.787062 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:18.787025    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerStarted","Data":"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"}
Sep 19 09:16:18.811609 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:18.811555    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podStartSLOduration=3.586315568 podStartE2EDuration="5.811539427s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.225223859s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:16:18.811362712 +0000 UTC m=+3330.997554669" watchObservedRunningTime="2026-09-19 09:16:18.811539427 +0000 UTC m=+3330.997731385"
Sep 19 09:16:23.738148 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:23.738102    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:23.738584 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:23.738191    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:23.739592 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:23.739559    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:16:33.738440 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:33.738385    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:16:33.758277 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:33.758237    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:16:43.738366 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:43.738306    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:16:45.769160 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:45.769123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:16:53.738304 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:16:53.738246    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:17:03.738024 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:17:03.737969    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:17:13.737624 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:17:13.737581    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:17:23.738439 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:17:23.738385    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:17:33.738363 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:17:33.738264    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:17:43.738229 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:17:43.738184    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:17:53.737536 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:17:53.737494    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:18:03.738472 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:03.738411    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8001/health\": dial tcp 10.134.0.73:8001: connect: connection refused"
Sep 19 09:18:13.747356 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:13.747318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:18:13.759059 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:13.759032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:18:23.249450 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:23.249412    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"]
Sep 19 09:18:23.253936 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:23.253905    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"]
Sep 19 09:18:25.769620 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:25.769581    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:18:35.768986 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:35.768942    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:18:38.270358 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:38.270313    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main" containerID="cri-o://883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051" gracePeriod=45
Sep 19 09:18:38.277320 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:38.277269    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main" containerID="cri-o://4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b" gracePeriod=45
Sep 19 09:18:38.889446 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:38.889411    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="llm-d-routing-sidecar" containerID="cri-o://beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b" gracePeriod=45
Sep 19 09:18:39.134821 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.134796    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:18:39.295388 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-kserve-provision-location\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v9sg\" (UniqueName: \"kubernetes.io/projected/cd4966ab-6e00-4b00-863e-ccb449203c88-kube-api-access-7v9sg\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295438    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-model-cache\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295463    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4966ab-6e00-4b00-863e-ccb449203c88-tls-certs\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295479    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd4966ab-6e00-4b00-863e-ccb449203c88-cabundle-cert\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-tmp-dir\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-dshm\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-home\") pod \"cd4966ab-6e00-4b00-863e-ccb449203c88\" (UID: \"cd4966ab-6e00-4b00-863e-ccb449203c88\") "
Sep 19 09:18:39.295812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295750    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.296291 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd4966ab-6e00-4b00-863e-ccb449203c88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:18:39.296291 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295974    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.296291 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.295991    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd4966ab-6e00-4b00-863e-ccb449203c88-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.296461 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.296410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-home" (OuterVolumeSpecName: "home") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.297901 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.297871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4966ab-6e00-4b00-863e-ccb449203c88-kube-api-access-7v9sg" (OuterVolumeSpecName: "kube-api-access-7v9sg") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "kube-api-access-7v9sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:18:39.298079 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.298062    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd4966ab-6e00-4b00-863e-ccb449203c88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:18:39.298176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.298156    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-dshm" (OuterVolumeSpecName: "dshm") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.308602 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.308576    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.318892 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.318865    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd4966ab-6e00-4b00-863e-ccb449203c88" (UID: "cd4966ab-6e00-4b00-863e-ccb449203c88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:18:39.396471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.396389    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.396471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.396419    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v9sg\" (UniqueName: \"kubernetes.io/projected/cd4966ab-6e00-4b00-863e-ccb449203c88-kube-api-access-7v9sg\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.396471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.396429    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd4966ab-6e00-4b00-863e-ccb449203c88-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.396471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.396438    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.396471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.396446    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.396471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.396454    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd4966ab-6e00-4b00-863e-ccb449203c88-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:18:39.451566 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451533    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerID="4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b" exitCode=0
Sep 19 09:18:39.451566 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451560    2580 generic.go:356] "Generic (PLEG): container finished" podID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerID="beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b" exitCode=0
Sep 19 09:18:39.451732 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerDied","Data":"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"}
Sep 19 09:18:39.451732 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerDied","Data":"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"}
Sep 19 09:18:39.451732 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92" event={"ID":"cd4966ab-6e00-4b00-863e-ccb449203c88","Type":"ContainerDied","Data":"5ea22635fb385088fad0e8b1bd29557f75428dc4388552c3bc2ea63f442e1683"}
Sep 19 09:18:39.451732 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451590    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"
Sep 19 09:18:39.451879 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.451670    2580 scope.go:117] "RemoveContainer" containerID="4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"
Sep 19 09:18:39.460894 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.460871    2580 scope.go:117] "RemoveContainer" containerID="95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"
Sep 19 09:18:39.468651 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.468633    2580 scope.go:117] "RemoveContainer" containerID="beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"
Sep 19 09:18:39.475295 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.475271    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"]
Sep 19 09:18:39.476791 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.476647    2580 scope.go:117] "RemoveContainer" containerID="4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"
Sep 19 09:18:39.477065 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:18:39.477037    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b\": container with ID starting with 4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b not found: ID does not exist" containerID="4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"
Sep 19 09:18:39.477143 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.477071    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"} err="failed to get container status \"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b\": rpc error: code = NotFound desc = could not find container \"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b\": container with ID starting with 4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b not found: ID does not exist"
Sep 19 09:18:39.477143 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.477095    2580 scope.go:117] "RemoveContainer" containerID="95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"
Sep 19 09:18:39.477448 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:18:39.477429    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd\": container with ID starting with 95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd not found: ID does not exist" containerID="95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"
Sep 19 09:18:39.477508 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.477456    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"} err="failed to get container status \"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd\": rpc error: code = NotFound desc = could not find container \"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd\": container with ID starting with 95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd not found: ID does not exist"
Sep 19 09:18:39.477508 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.477478    2580 scope.go:117] "RemoveContainer" containerID="beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"
Sep 19 09:18:39.477790 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:18:39.477770    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b\": container with ID starting with beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b not found: ID does not exist" containerID="beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"
Sep 19 09:18:39.477876 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.477796    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"} err="failed to get container status \"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b\": rpc error: code = NotFound desc = could not find container \"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b\": container with ID starting with beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b not found: ID does not exist"
Sep 19 09:18:39.477876 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.477810    2580 scope.go:117] "RemoveContainer" containerID="4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"
Sep 19 09:18:39.478051 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.478036    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b"} err="failed to get container status \"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b\": rpc error: code = NotFound desc = could not find container \"4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b\": container with ID starting with 4642817a7c9efdf299b983cad65927059f345a004d70ef924dcfe8e2c31c292b not found: ID does not exist"
Sep 19 09:18:39.478100 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.478051    2580 scope.go:117] "RemoveContainer" containerID="95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"
Sep 19 09:18:39.478302 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.478277    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd"} err="failed to get container status \"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd\": rpc error: code = NotFound desc = could not find container \"95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd\": container with ID starting with 95d33c891a5fba44e2a8d92b6dec3b41cdbdaf640da0614afebe3ebbbadf13fd not found: ID does not exist"
Sep 19 09:18:39.478391 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.478304    2580 scope.go:117] "RemoveContainer" containerID="beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"
Sep 19 09:18:39.478590 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.478572    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b"} err="failed to get container status \"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b\": rpc error: code = NotFound desc = could not find container \"beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b\": container with ID starting with beed79018d5c841f46163bed53934a00ee8240658956efbd933bed7b3cdc495b not found: ID does not exist"
Sep 19 09:18:39.479053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:39.479034    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-6757b868d4-sjv92"]
Sep 19 09:18:40.079864 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:40.079828    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" path="/var/lib/kubelet/pods/cd4966ab-6e00-4b00-863e-ccb449203c88/volumes"
Sep 19 09:18:45.769380 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:45.769337    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:18:45.769878 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:45.769467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:18:55.769280 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:18:55.769235    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:19:05.768807 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:05.768716    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:19:08.513711 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.513682    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:19:08.581398 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.581305    2580 generic.go:356] "Generic (PLEG): container finished" podID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerID="883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051" exitCode=0
Sep 19 09:19:08.581398 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.581349    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"
Sep 19 09:19:08.581589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.581352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" event={"ID":"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f","Type":"ContainerDied","Data":"883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051"}
Sep 19 09:19:08.581589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.581474    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt" event={"ID":"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f","Type":"ContainerDied","Data":"72169e72c005dd8765117a57e557869581f941eca3d2604fae092eab9f14b7ee"}
Sep 19 09:19:08.581589 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.581501    2580 scope.go:117] "RemoveContainer" containerID="883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051"
Sep 19 09:19:08.589582 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.589565    2580 scope.go:117] "RemoveContainer" containerID="883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051"
Sep 19 09:19:08.589859 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:19:08.589829    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051\": container with ID starting with 883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051 not found: ID does not exist" containerID="883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051"
Sep 19 09:19:08.589966 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.589865    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051"} err="failed to get container status \"883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051\": rpc error: code = NotFound desc = could not find container \"883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051\": container with ID starting with 883d51e3402e4220e49e8aeb34275f5402f9cb0ba5c6708910848c0e154bc051 not found: ID does not exist"
Sep 19 09:19:08.630598 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.630567    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-tls-certs\") pod \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") "
Sep 19 09:19:08.630753 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.630661    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh8nk\" (UniqueName: \"kubernetes.io/projected/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-kube-api-access-nh8nk\") pod \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\" (UID: \"f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f\") "
Sep 19 09:19:08.632622 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.632598    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" (UID: "f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:19:08.632736 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.632662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-kube-api-access-nh8nk" (OuterVolumeSpecName: "kube-api-access-nh8nk") pod "f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" (UID: "f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f"). InnerVolumeSpecName "kube-api-access-nh8nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:19:08.732027 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.731989    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:08.732027 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.732021    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8nk\" (UniqueName: \"kubernetes.io/projected/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f-kube-api-access-nh8nk\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:08.906373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.906337    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"]
Sep 19 09:19:08.911342 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:08.911318    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zwpvt"]
Sep 19 09:19:10.079512 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:10.079479    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" path="/var/lib/kubelet/pods/f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f/volumes"
Sep 19 09:19:18.287509 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:18.287469    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"]
Sep 19 09:19:18.288080 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:18.287670    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy" containerID="cri-o://7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d" gracePeriod=30
Sep 19 09:19:21.190635 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190594    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"]
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190936    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="llm-d-routing-sidecar"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190950    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="llm-d-routing-sidecar"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190959    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190964    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190978    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190983    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190993    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="storage-initializer"
Sep 19 09:19:21.191053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.190999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="storage-initializer"
Sep 19 09:19:21.191339 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.191060    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="main"
Sep 19 09:19:21.191339 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.191070    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4966ab-6e00-4b00-863e-ccb449203c88" containerName="llm-d-routing-sidecar"
Sep 19 09:19:21.191339 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.191079    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8f302ea-f9e0-4fd0-b6fa-ff678dd1db4f" containerName="main"
Sep 19 09:19:21.195613 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.195594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.199417 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.199340    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:19:21.199417 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.199320    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 19 09:19:21.199417 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.199410    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-nsfg8\""
Sep 19 09:19:21.199761 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.199737    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 19 09:19:21.199886 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.199743    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 19 09:19:21.206093 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.206059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 19 09:19:21.208832 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.208811    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"]
Sep 19 09:19:21.343142 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2769fc2-ba09-4969-b905-efddc30e56a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343142 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343184    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql8pc\" (UniqueName: \"kubernetes.io/projected/a2769fc2-ba09-4969-b905-efddc30e56a7-kube-api-access-ql8pc\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343387    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2769fc2-ba09-4969-b905-efddc30e56a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.343594 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.343439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444604 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql8pc\" (UniqueName: \"kubernetes.io/projected/a2769fc2-ba09-4969-b905-efddc30e56a7-kube-api-access-ql8pc\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2769fc2-ba09-4969-b905-efddc30e56a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2769fc2-ba09-4969-b905-efddc30e56a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.444799 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.444774    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.445479 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.445063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.445479 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.445083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.445479 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.445140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.445479 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.445349    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.445637 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.445558    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2769fc2-ba09-4969-b905-efddc30e56a7-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.447201 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.447182    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.447340 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.447324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2769fc2-ba09-4969-b905-efddc30e56a7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.454049 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.454027    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql8pc\" (UniqueName: \"kubernetes.io/projected/a2769fc2-ba09-4969-b905-efddc30e56a7-kube-api-access-ql8pc\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.506013 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.505980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:21.784762 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:21.784564    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"]
Sep 19 09:19:21.788050 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:19:21.788005    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2769fc2_ba09_4969_b905_efddc30e56a7.slice/crio-5277ccb644d4001790c0445602cf78216b46f114a8f3d35e94dff8c2a58d4ca0 WatchSource:0}: Error finding container 5277ccb644d4001790c0445602cf78216b46f114a8f3d35e94dff8c2a58d4ca0: Status 404 returned error can't find the container with id 5277ccb644d4001790c0445602cf78216b46f114a8f3d35e94dff8c2a58d4ca0
Sep 19 09:19:22.652255 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:22.652186    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" event={"ID":"a2769fc2-ba09-4969-b905-efddc30e56a7","Type":"ContainerStarted","Data":"ec99d4d6c890d945e3d3e567df65b8f42d0946a6361c0a378babe5d52402b03b"}
Sep 19 09:19:22.652765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:22.652295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" event={"ID":"a2769fc2-ba09-4969-b905-efddc30e56a7","Type":"ContainerStarted","Data":"5277ccb644d4001790c0445602cf78216b46f114a8f3d35e94dff8c2a58d4ca0"}
Sep 19 09:19:23.544010 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.543981    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:19:23.660194 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.660151    2580 generic.go:356] "Generic (PLEG): container finished" podID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerID="7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d" exitCode=0
Sep 19 09:19:23.660730 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.660232    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"
Sep 19 09:19:23.660730 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.660283    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" event={"ID":"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da","Type":"ContainerDied","Data":"7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d"}
Sep 19 09:19:23.660730 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.660327    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr" event={"ID":"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da","Type":"ContainerDied","Data":"e28cb3812d0633c84fc9315706436c93b9ea6b39426ff971baa77f3dd9e3b4a0"}
Sep 19 09:19:23.660730 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.660356    2580 scope.go:117] "RemoveContainer" containerID="7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d"
Sep 19 09:19:23.665810 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.665786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-envoy\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.665930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.665833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-token\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.665930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.665860    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-podinfo\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.665930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.665880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istiod-ca-cert\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.665930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.665913    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnspx\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-kube-api-access-bnspx\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.666156 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.665963    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-socket\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.666156 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666018    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-credential-socket\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.666156 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-data\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.666156 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-certs\") pod \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\" (UID: \"6a3ae3d1-1da5-4aaa-9b72-ba999cec60da\") "
Sep 19 09:19:23.666411 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666384    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:19:23.666471 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666425    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:19:23.666684 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666664    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:19:23.666787 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666744    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:19:23.667121 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.666922    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:19:23.668996 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.668968    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:19:23.669260 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.669207    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 19 09:19:23.669348 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.669273    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-kube-api-access-bnspx" (OuterVolumeSpecName: "kube-api-access-bnspx") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "kube-api-access-bnspx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:19:23.669765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.669729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" (UID: "6a3ae3d1-1da5-4aaa-9b72-ba999cec60da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:19:23.671331 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.671307    2580 scope.go:117] "RemoveContainer" containerID="cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"
Sep 19 09:19:23.688194 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.688165    2580 scope.go:117] "RemoveContainer" containerID="7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d"
Sep 19 09:19:23.688599 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:19:23.688577    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d\": container with ID starting with 7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d not found: ID does not exist" containerID="7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d"
Sep 19 09:19:23.688675 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.688609    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d"} err="failed to get container status \"7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d\": rpc error: code = NotFound desc = could not find container \"7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d\": container with ID starting with 7502adede0feb8f7be258a49800c9822baad38012315f11472c5c9e8c1e1019d not found: ID does not exist"
Sep 19 09:19:23.688675 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.688628    2580 scope.go:117] "RemoveContainer" containerID="cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"
Sep 19 09:19:23.688884 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:19:23.688865    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e\": container with ID starting with cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e not found: ID does not exist" containerID="cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"
Sep 19 09:19:23.688936 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.688888    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e"} err="failed to get container status \"cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e\": rpc error: code = NotFound desc = could not find container \"cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e\": container with ID starting with cd84c1554447516d4747c0298218510e95f12dda32865b8dee73631812ad267e not found: ID does not exist"
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767264    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767304    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-envoy\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767318    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-token\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767330    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-podinfo\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767342    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istiod-ca-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767357    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnspx\" (UniqueName: \"kubernetes.io/projected/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-kube-api-access-bnspx\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767371    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-workload-socket\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767907 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767383    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-credential-socket\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.767907 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.767396    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da-istio-data\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:19:23.992774 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.992732    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"]
Sep 19 09:19:23.996718 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:23.996684    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-fgcjr"]
Sep 19 09:19:24.080843 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:24.080801    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" path="/var/lib/kubelet/pods/6a3ae3d1-1da5-4aaa-9b72-ba999cec60da/volumes"
Sep 19 09:19:24.666916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:24.666878    2580 generic.go:356] "Generic (PLEG): container finished" podID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerID="ec99d4d6c890d945e3d3e567df65b8f42d0946a6361c0a378babe5d52402b03b" exitCode=0
Sep 19 09:19:24.667335 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:24.666951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" event={"ID":"a2769fc2-ba09-4969-b905-efddc30e56a7","Type":"ContainerDied","Data":"ec99d4d6c890d945e3d3e567df65b8f42d0946a6361c0a378babe5d52402b03b"}
Sep 19 09:19:25.673142 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:25.673109    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" event={"ID":"a2769fc2-ba09-4969-b905-efddc30e56a7","Type":"ContainerStarted","Data":"63342ed82a1ead140b07eb8da0743818104c795798208ef1b90f58e8f8dd29a1"}
Sep 19 09:19:25.697063 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:25.697007    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podStartSLOduration=2.46872103 podStartE2EDuration="4.69699156s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.22827053s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:19:25.695297261 +0000 UTC m=+3517.881489220" watchObservedRunningTime="2026-09-19 09:19:25.69699156 +0000 UTC m=+3517.883183517"
Sep 19 09:19:26.507083 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:26.507045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:31.506456 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:31.506407    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:19:31.508045 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:31.508014    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:19:41.506457 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:41.506403    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:19:51.506703 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:19:51.506655    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:20:01.507430 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:01.507373    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:20:11.506655 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:11.506606    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:20:21.506745 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:21.506697    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:20:31.506767 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:31.506716    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:20:32.007643 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.007608    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"]
Sep 19 09:20:32.007986 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.007966    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy"
Sep 19 09:20:32.008032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.007992    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy"
Sep 19 09:20:32.008032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.008007    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy"
Sep 19 09:20:32.008032 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.008015    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy"
Sep 19 09:20:32.008133 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.008094    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy"
Sep 19 09:20:32.008133 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.008102    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3ae3d1-1da5-4aaa-9b72-ba999cec60da" containerName="istio-proxy"
Sep 19 09:20:32.010985 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.010951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.014990 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.014960    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 19 09:20:32.015134 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.015059    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-pwr2h\""
Sep 19 09:20:32.015134 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.015100    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 19 09:20:32.015285 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.015247    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 19 09:20:32.015388 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.015369    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-xgcgl\""
Sep 19 09:20:32.020792 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.020764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"]
Sep 19 09:20:32.143164 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.143053    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vnv\" (UniqueName: \"kubernetes.io/projected/233330e8-ee83-4e6f-a47b-2ced2b497cda-kube-api-access-w7vnv\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.143164 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.143114    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233330e8-ee83-4e6f-a47b-2ced2b497cda-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.243741 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.243706    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233330e8-ee83-4e6f-a47b-2ced2b497cda-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.243898 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.243791    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vnv\" (UniqueName: \"kubernetes.io/projected/233330e8-ee83-4e6f-a47b-2ced2b497cda-kube-api-access-w7vnv\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.246507 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.246481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233330e8-ee83-4e6f-a47b-2ced2b497cda-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.251822 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.251797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vnv\" (UniqueName: \"kubernetes.io/projected/233330e8-ee83-4e6f-a47b-2ced2b497cda-kube-api-access-w7vnv\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.323821 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.323718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:32.598342 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.598309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"]
Sep 19 09:20:32.600167 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:20:32.600129    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233330e8_ee83_4e6f_a47b_2ced2b497cda.slice/crio-79bb9c566e80236d3d1309c5dde198d54b66f73a0ceb7fd009489ad6c4dfdcd9 WatchSource:0}: Error finding container 79bb9c566e80236d3d1309c5dde198d54b66f73a0ceb7fd009489ad6c4dfdcd9: Status 404 returned error can't find the container with id 79bb9c566e80236d3d1309c5dde198d54b66f73a0ceb7fd009489ad6c4dfdcd9
Sep 19 09:20:32.981133 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.981090    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" event={"ID":"233330e8-ee83-4e6f-a47b-2ced2b497cda","Type":"ContainerStarted","Data":"65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1"}
Sep 19 09:20:32.981360 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.981191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" event={"ID":"233330e8-ee83-4e6f-a47b-2ced2b497cda","Type":"ContainerStarted","Data":"79bb9c566e80236d3d1309c5dde198d54b66f73a0ceb7fd009489ad6c4dfdcd9"}
Sep 19 09:20:32.981360 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:32.981280    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:20:33.002114 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:33.002045    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" podStartSLOduration=2.002030067 podStartE2EDuration="2.002030067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:20:33.001349772 +0000 UTC m=+3585.187541730" watchObservedRunningTime="2026-09-19 09:20:33.002030067 +0000 UTC m=+3585.188222023"
Sep 19 09:20:41.506947 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:41.506894    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:20:51.506857 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:20:51.506800    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:21:01.506772 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:01.506711    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:21:03.987138 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:03.987108    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:21:11.507381 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:11.507324    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": dial tcp 10.134.0.75:8000: connect: connection refused"
Sep 19 09:21:21.517524 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:21.517488    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:21:21.525214 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:21.525188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:21:25.577777 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:25.577741    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"]
Sep 19 09:21:40.599602 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:40.599540    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main" containerID="cri-o://63342ed82a1ead140b07eb8da0743818104c795798208ef1b90f58e8f8dd29a1" gracePeriod=45
Sep 19 09:21:41.291936 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.291900    2580 generic.go:356] "Generic (PLEG): container finished" podID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerID="63342ed82a1ead140b07eb8da0743818104c795798208ef1b90f58e8f8dd29a1" exitCode=0
Sep 19 09:21:41.292090 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.291965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" event={"ID":"a2769fc2-ba09-4969-b905-efddc30e56a7","Type":"ContainerDied","Data":"63342ed82a1ead140b07eb8da0743818104c795798208ef1b90f58e8f8dd29a1"}
Sep 19 09:21:41.342182 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.342157    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:21:41.429695 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-model-cache\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429695 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2769fc2-ba09-4969-b905-efddc30e56a7-tls-certs\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429697    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-dshm\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-kserve-provision-location\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429745    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql8pc\" (UniqueName: \"kubernetes.io/projected/a2769fc2-ba09-4969-b905-efddc30e56a7-kube-api-access-ql8pc\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-home\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2769fc2-ba09-4969-b905-efddc30e56a7-cabundle-cert\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.429930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429794    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-tmp-dir\") pod \"a2769fc2-ba09-4969-b905-efddc30e56a7\" (UID: \"a2769fc2-ba09-4969-b905-efddc30e56a7\") "
Sep 19 09:21:41.430215 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.429934    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:21:41.430215 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.430070    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.430359 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.430329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2769fc2-ba09-4969-b905-efddc30e56a7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:21:41.430833 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.430804    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-home" (OuterVolumeSpecName: "home") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:21:41.432159 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.432128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-dshm" (OuterVolumeSpecName: "dshm") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:21:41.432159 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.432128    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2769fc2-ba09-4969-b905-efddc30e56a7-kube-api-access-ql8pc" (OuterVolumeSpecName: "kube-api-access-ql8pc") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "kube-api-access-ql8pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:21:41.432336 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.432233    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2769fc2-ba09-4969-b905-efddc30e56a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:21:41.449005 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.448960    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:21:41.455254 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.455214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2769fc2-ba09-4969-b905-efddc30e56a7" (UID: "a2769fc2-ba09-4969-b905-efddc30e56a7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:21:41.531035 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531006    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2769fc2-ba09-4969-b905-efddc30e56a7-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.531035 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531030    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.531035 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531040    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.531299 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531049    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql8pc\" (UniqueName: \"kubernetes.io/projected/a2769fc2-ba09-4969-b905-efddc30e56a7-kube-api-access-ql8pc\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.531299 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531057    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.531299 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531067    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2769fc2-ba09-4969-b905-efddc30e56a7-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:41.531299 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:41.531076    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2769fc2-ba09-4969-b905-efddc30e56a7-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:21:42.298598 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:42.298559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr" event={"ID":"a2769fc2-ba09-4969-b905-efddc30e56a7","Type":"ContainerDied","Data":"5277ccb644d4001790c0445602cf78216b46f114a8f3d35e94dff8c2a58d4ca0"}
Sep 19 09:21:42.298598 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:42.298583    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"
Sep 19 09:21:42.299117 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:42.298632    2580 scope.go:117] "RemoveContainer" containerID="63342ed82a1ead140b07eb8da0743818104c795798208ef1b90f58e8f8dd29a1"
Sep 19 09:21:42.307105 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:42.307085    2580 scope.go:117] "RemoveContainer" containerID="ec99d4d6c890d945e3d3e567df65b8f42d0946a6361c0a378babe5d52402b03b"
Sep 19 09:21:42.317187 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:42.317124    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"]
Sep 19 09:21:42.321310 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:42.321289    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6f65b94475fbrmr"]
Sep 19 09:21:44.079313 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:44.079280    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" path="/var/lib/kubelet/pods/a2769fc2-ba09-4969-b905-efddc30e56a7/volumes"
Sep 19 09:21:58.638606 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.638561    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 19 09:21:58.639116 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.638913    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="storage-initializer"
Sep 19 09:21:58.639116 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.638927    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="storage-initializer"
Sep 19 09:21:58.639116 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.638938    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main"
Sep 19 09:21:58.639116 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.638943    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main"
Sep 19 09:21:58.639116 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.638999    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2769fc2-ba09-4969-b905-efddc30e56a7" containerName="main"
Sep 19 09:21:58.641893 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.641875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.644642 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.644612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 19 09:21:58.644810 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.644677    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 19 09:21:58.644873 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.644823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-6qr9j\""
Sep 19 09:21:58.645776 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.645756    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 19 09:21:58.645861 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.645788    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-dm7px\""
Sep 19 09:21:58.645924 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.645793    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 19 09:21:58.645924 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.645767    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 19 09:21:58.655713 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.655691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 19 09:21:58.787448 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhg2\" (UniqueName: \"kubernetes.io/projected/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kube-api-access-lnhg2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787448 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787674 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787674 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787674 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787674 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787625    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4224e7-c07d-4aa6-ae4a-bf2131356692-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787674 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.787843 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.787681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.888741 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888636    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhg2\" (UniqueName: \"kubernetes.io/projected/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kube-api-access-lnhg2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.888741 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.888741 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888699    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888821    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4224e7-c07d-4aa6-ae4a-bf2131356692-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.888991    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889310 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.889085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889310 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.889104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889565 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.889545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889642 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.889593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.889873 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.889851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4224e7-c07d-4aa6-ae4a-bf2131356692-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.891250 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.891211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.891438 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.891423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.899122 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.899100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhg2\" (UniqueName: \"kubernetes.io/projected/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kube-api-access-lnhg2\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:58.952660 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:58.952623    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:21:59.233285 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:59.233056    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 19 09:21:59.237433 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:21:59.237396    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4224e7_c07d_4aa6_ae4a_bf2131356692.slice/crio-91a3342e4c6db843b5473af92225d62b90fafdef5a00af339807c3525b44a3a1 WatchSource:0}: Error finding container 91a3342e4c6db843b5473af92225d62b90fafdef5a00af339807c3525b44a3a1: Status 404 returned error can't find the container with id 91a3342e4c6db843b5473af92225d62b90fafdef5a00af339807c3525b44a3a1
Sep 19 09:21:59.370771 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:59.370737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3f4224e7-c07d-4aa6-ae4a-bf2131356692","Type":"ContainerStarted","Data":"044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae"}
Sep 19 09:21:59.370965 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:21:59.370802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3f4224e7-c07d-4aa6-ae4a-bf2131356692","Type":"ContainerStarted","Data":"91a3342e4c6db843b5473af92225d62b90fafdef5a00af339807c3525b44a3a1"}
Sep 19 09:22:02.387961 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:02.387875    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerID="044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae" exitCode=0
Sep 19 09:22:02.387961 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:02.387947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3f4224e7-c07d-4aa6-ae4a-bf2131356692","Type":"ContainerDied","Data":"044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae"}
Sep 19 09:22:03.394769 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:03.394738    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3f4224e7-c07d-4aa6-ae4a-bf2131356692","Type":"ContainerStarted","Data":"b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7"}
Sep 19 09:22:03.416646 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:03.416599    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.1728337030000002 podStartE2EDuration="5.416584213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.24375051s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:22:03.414810178 +0000 UTC m=+3675.601002146" watchObservedRunningTime="2026-09-19 09:22:03.416584213 +0000 UTC m=+3675.602776170"
Sep 19 09:22:03.953590 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:03.953554    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:22:08.953386 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:08.953346    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:22:08.954847 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:08.954813    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:22:18.953165 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:18.953109    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:22:28.953559 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:28.953515    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:22:37.727413 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:37.727372    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"]
Sep 19 09:22:38.953843 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:38.953789    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:22:43.987294 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:43.987251    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:22:48.953794 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:48.953744    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:22:52.749554 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:52.749507    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main" containerID="cri-o://65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1" gracePeriod=45
Sep 19 09:22:53.987337 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:53.987288    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:22:58.953871 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:22:58.953819    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:23:03.986948 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:03.986906    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:23:03.987501 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:03.987021    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:23:08.953666 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:08.953610    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:23:13.986953 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:13.986914    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:23:18.953344 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:18.953300    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:23:23.004659 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.004630    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:23:23.027714 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.027632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233330e8-ee83-4e6f-a47b-2ced2b497cda-tls-certs\") pod \"233330e8-ee83-4e6f-a47b-2ced2b497cda\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") "
Sep 19 09:23:23.027714 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.027675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7vnv\" (UniqueName: \"kubernetes.io/projected/233330e8-ee83-4e6f-a47b-2ced2b497cda-kube-api-access-w7vnv\") pod \"233330e8-ee83-4e6f-a47b-2ced2b497cda\" (UID: \"233330e8-ee83-4e6f-a47b-2ced2b497cda\") "
Sep 19 09:23:23.030568 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.030534    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233330e8-ee83-4e6f-a47b-2ced2b497cda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "233330e8-ee83-4e6f-a47b-2ced2b497cda" (UID: "233330e8-ee83-4e6f-a47b-2ced2b497cda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:23:23.030708 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.030544    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233330e8-ee83-4e6f-a47b-2ced2b497cda-kube-api-access-w7vnv" (OuterVolumeSpecName: "kube-api-access-w7vnv") pod "233330e8-ee83-4e6f-a47b-2ced2b497cda" (UID: "233330e8-ee83-4e6f-a47b-2ced2b497cda"). InnerVolumeSpecName "kube-api-access-w7vnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:23:23.129244 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.129179    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233330e8-ee83-4e6f-a47b-2ced2b497cda-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:23:23.129244 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.129238    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7vnv\" (UniqueName: \"kubernetes.io/projected/233330e8-ee83-4e6f-a47b-2ced2b497cda-kube-api-access-w7vnv\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:23:23.758829 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.758785    2580 generic.go:356] "Generic (PLEG): container finished" podID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerID="65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1" exitCode=0
Sep 19 09:23:23.759046 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.758861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" event={"ID":"233330e8-ee83-4e6f-a47b-2ced2b497cda","Type":"ContainerDied","Data":"65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1"}
Sep 19 09:23:23.759046 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.758907    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"
Sep 19 09:23:23.759046 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.758919    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7" event={"ID":"233330e8-ee83-4e6f-a47b-2ced2b497cda","Type":"ContainerDied","Data":"79bb9c566e80236d3d1309c5dde198d54b66f73a0ceb7fd009489ad6c4dfdcd9"}
Sep 19 09:23:23.759046 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.758952    2580 scope.go:117] "RemoveContainer" containerID="65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1"
Sep 19 09:23:23.768896 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.768876    2580 scope.go:117] "RemoveContainer" containerID="65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1"
Sep 19 09:23:23.769252 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:23:23.769209    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1\": container with ID starting with 65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1 not found: ID does not exist" containerID="65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1"
Sep 19 09:23:23.769312 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.769262    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1"} err="failed to get container status \"65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1\": rpc error: code = NotFound desc = could not find container \"65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1\": container with ID starting with 65d1ef6db6055f46e5566702fad40aa4e53d41abae68105c6001f2df965902c1 not found: ID does not exist"
Sep 19 09:23:23.780336 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.780307    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"]
Sep 19 09:23:23.784973 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:23.784936    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-fgxh7"]
Sep 19 09:23:24.079891 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:24.079805    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" path="/var/lib/kubelet/pods/233330e8-ee83-4e6f-a47b-2ced2b497cda/volumes"
Sep 19 09:23:28.953735 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:28.953693    2580 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:23:34.322155 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.322076    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"]
Sep 19 09:23:34.322557 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.322393    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main"
Sep 19 09:23:34.322557 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.322406    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main"
Sep 19 09:23:34.322557 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.322470    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="233330e8-ee83-4e6f-a47b-2ced2b497cda" containerName="main"
Sep 19 09:23:34.326785 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.326764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.330147 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.330095    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 19 09:23:34.330147 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.330103    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 19 09:23:34.330376 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.330152    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-tjwkk\""
Sep 19 09:23:34.330376 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.330141    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-lvj6g\""
Sep 19 09:23:34.330376 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.330258    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 19 09:23:34.335156 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.335128    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"]
Sep 19 09:23:34.430824 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.430788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3bed6b-3c89-4175-85ed-91150def9f8a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.430997 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.430922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f5fg\" (UniqueName: \"kubernetes.io/projected/0d3bed6b-3c89-4175-85ed-91150def9f8a-kube-api-access-7f5fg\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.532057 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.532024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f5fg\" (UniqueName: \"kubernetes.io/projected/0d3bed6b-3c89-4175-85ed-91150def9f8a-kube-api-access-7f5fg\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.532268 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.532072    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3bed6b-3c89-4175-85ed-91150def9f8a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.534545 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.534525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3bed6b-3c89-4175-85ed-91150def9f8a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.540267 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.540241    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f5fg\" (UniqueName: \"kubernetes.io/projected/0d3bed6b-3c89-4175-85ed-91150def9f8a-kube-api-access-7f5fg\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.637419 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.637323    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:34.917030 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:34.917003    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"]
Sep 19 09:23:34.920491 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:23:34.920449    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3bed6b_3c89_4175_85ed_91150def9f8a.slice/crio-d0821a8d62ed994f1e852b3e79a0337c46ed13434d8350c3e21452f7032e2ade WatchSource:0}: Error finding container d0821a8d62ed994f1e852b3e79a0337c46ed13434d8350c3e21452f7032e2ade: Status 404 returned error can't find the container with id d0821a8d62ed994f1e852b3e79a0337c46ed13434d8350c3e21452f7032e2ade
Sep 19 09:23:35.816903 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:35.816867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" event={"ID":"0d3bed6b-3c89-4175-85ed-91150def9f8a","Type":"ContainerStarted","Data":"86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107"}
Sep 19 09:23:35.817364 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:35.816918    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" event={"ID":"0d3bed6b-3c89-4175-85ed-91150def9f8a","Type":"ContainerStarted","Data":"d0821a8d62ed994f1e852b3e79a0337c46ed13434d8350c3e21452f7032e2ade"}
Sep 19 09:23:35.817364 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:35.816964    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:23:35.835552 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:35.835497    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" podStartSLOduration=1.8354804169999999 podStartE2EDuration="1.835480417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:23:35.835063466 +0000 UTC m=+3768.021255425" watchObservedRunningTime="2026-09-19 09:23:35.835480417 +0000 UTC m=+3768.021672374"
Sep 19 09:23:38.962690 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:38.962659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:23:38.970604 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:38.970580    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:23:45.830507 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:23:45.830470    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 19 09:24:00.854190 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:00.854143    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" containerID="cri-o://b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7" gracePeriod=45
Sep 19 09:24:00.963277 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:00.963206    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 19 09:24:01.594056 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.594030    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:24:01.766302 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-model-cache\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766318    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tls-certs\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766343    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4224e7-c07d-4aa6-ae4a-bf2131356692-cabundle-cert\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tmp-dir\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766411    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kserve-provision-location\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766482 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnhg2\" (UniqueName: \"kubernetes.io/projected/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kube-api-access-lnhg2\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766492    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-home\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766519    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-dshm\") pod \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\" (UID: \"3f4224e7-c07d-4aa6-ae4a-bf2131356692\") "
Sep 19 09:24:01.766765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766667    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:01.766921 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766797    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.766921 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.766798    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4224e7-c07d-4aa6-ae4a-bf2131356692-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 19 09:24:01.767503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.767471    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-home" (OuterVolumeSpecName: "home") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:01.768941 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.768912    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:24:01.768941 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.768926    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kube-api-access-lnhg2" (OuterVolumeSpecName: "kube-api-access-lnhg2") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "kube-api-access-lnhg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:24:01.769122 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.769005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-dshm" (OuterVolumeSpecName: "dshm") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:01.785048 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.785009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:01.796520 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.796489    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f4224e7-c07d-4aa6-ae4a-bf2131356692" (UID: "3f4224e7-c07d-4aa6-ae4a-bf2131356692"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:24:01.867795 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867760    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.867795 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867788    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f4224e7-c07d-4aa6-ae4a-bf2131356692-cabundle-cert\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.867795 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867798    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.868286 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867807    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kserve-provision-location\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.868286 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867816    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhg2\" (UniqueName: \"kubernetes.io/projected/3f4224e7-c07d-4aa6-ae4a-bf2131356692-kube-api-access-lnhg2\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.868286 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867824    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.868286 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.867833    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f4224e7-c07d-4aa6-ae4a-bf2131356692-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:24:01.941418 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.941379    2580 generic.go:356] "Generic (PLEG): container finished" podID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerID="b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7" exitCode=0
Sep 19 09:24:01.941586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.941440    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Sep 19 09:24:01.941586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.941459    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3f4224e7-c07d-4aa6-ae4a-bf2131356692","Type":"ContainerDied","Data":"b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7"}
Sep 19 09:24:01.941586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.941526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3f4224e7-c07d-4aa6-ae4a-bf2131356692","Type":"ContainerDied","Data":"91a3342e4c6db843b5473af92225d62b90fafdef5a00af339807c3525b44a3a1"}
Sep 19 09:24:01.941586 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.941549    2580 scope.go:117] "RemoveContainer" containerID="b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7"
Sep 19 09:24:01.950162 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.950144    2580 scope.go:117] "RemoveContainer" containerID="044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae"
Sep 19 09:24:01.957622 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.957602    2580 scope.go:117] "RemoveContainer" containerID="b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7"
Sep 19 09:24:01.957878 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:24:01.957859    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7\": container with ID starting with b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7 not found: ID does not exist" containerID="b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7"
Sep 19 09:24:01.957935 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.957888    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7"} err="failed to get container status \"b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7\": rpc error: code = NotFound desc = could not find container \"b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7\": container with ID starting with b841645d3ddd1a6ed2601c0a9412207b7fb3d655ee0752099cf294e1037956a7 not found: ID does not exist"
Sep 19 09:24:01.957935 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.957904    2580 scope.go:117] "RemoveContainer" containerID="044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae"
Sep 19 09:24:01.958158 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:24:01.958139    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae\": container with ID starting with 044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae not found: ID does not exist" containerID="044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae"
Sep 19 09:24:01.958230 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.958167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae"} err="failed to get container status \"044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae\": rpc error: code = NotFound desc = could not find container \"044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae\": container with ID starting with 044c4bc26b24c2637678330bfa3ebb2f82538b2f7bd75d8cad2c0e0a0e58d4ae not found: ID does not exist"
Sep 19 09:24:01.963736 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.963713    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 19 09:24:01.967166 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:01.967144    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Sep 19 09:24:02.079629 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:02.079540    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" path="/var/lib/kubelet/pods/3f4224e7-c07d-4aa6-ae4a-bf2131356692/volumes"
Sep 19 09:24:06.823655 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:24:06.823625    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:25:04.580367 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.580277    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"]
Sep 19 09:25:04.580844 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.580615    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main"
Sep 19 09:25:04.580844 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.580627    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main"
Sep 19 09:25:04.580844 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.580645    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="storage-initializer"
Sep 19 09:25:04.580844 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.580652    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="storage-initializer"
Sep 19 09:25:04.580844 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.580716    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f4224e7-c07d-4aa6-ae4a-bf2131356692" containerName="main"
Sep 19 09:25:04.583521 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.583502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.587183 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.587158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-4g98g\""
Sep 19 09:25:04.587183 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.587181    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 19 09:25:04.587409 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.587168    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 19 09:25:04.587409 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.587249    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 19 09:25:04.595204 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.595177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"]
Sep 19 09:25:04.772975 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.772929    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.772975 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.772974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.773196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.773009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.773196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.773050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zw2\" (UniqueName: \"kubernetes.io/projected/bfbe2120-7216-42cc-bc62-421846081679-kube-api-access-v9zw2\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.773196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.773110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbe2120-7216-42cc-bc62-421846081679-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.773196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.773138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.806698 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.806662    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"]
Sep 19 09:25:04.809937 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.809914    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:04.812485 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.812462    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-jxc6k\""
Sep 19 09:25:04.819330 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.819304    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"]
Sep 19 09:25:04.873784 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.873702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zw2\" (UniqueName: \"kubernetes.io/projected/bfbe2120-7216-42cc-bc62-421846081679-kube-api-access-v9zw2\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.873784 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.873760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbe2120-7216-42cc-bc62-421846081679-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.873997 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.873797    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.873997 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.873962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.874113 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.873998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.874113 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.874016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.874211 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.874192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.874381 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.874353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.874537 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.874356    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.876171 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.876142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.876429 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.876409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbe2120-7216-42cc-bc62-421846081679-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.881816 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.881794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zw2\" (UniqueName: \"kubernetes.io/projected/bfbe2120-7216-42cc-bc62-421846081679-kube-api-access-v9zw2\") pod \"scheduler-inline-config-test-kserve-8655bc757-dsqbj\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.894654 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.894634    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:04.974894 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.974808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f799ce05-e5d6-42e4-b145-0a4a534f84b8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbsst22\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:04.974894 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:04.974878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdh9\" (UniqueName: \"kubernetes.io/projected/f799ce05-e5d6-42e4-b145-0a4a534f84b8-kube-api-access-bqdh9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbsst22\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:05.075886 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.075844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f799ce05-e5d6-42e4-b145-0a4a534f84b8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbsst22\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:05.076080 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.075929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdh9\" (UniqueName: \"kubernetes.io/projected/f799ce05-e5d6-42e4-b145-0a4a534f84b8-kube-api-access-bqdh9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbsst22\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:05.078645 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.078618    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f799ce05-e5d6-42e4-b145-0a4a534f84b8-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbsst22\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:05.086029 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.086001    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdh9\" (UniqueName: \"kubernetes.io/projected/f799ce05-e5d6-42e4-b145-0a4a534f84b8-kube-api-access-bqdh9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbsst22\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:05.120548 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.120523    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:05.166158 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.166129    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"]
Sep 19 09:25:05.169407 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:25:05.169377    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbe2120_7216_42cc_bc62_421846081679.slice/crio-943a70a918926cf3a0697d69cc965e4206d5faf6ded18ba2bea4132005e03652 WatchSource:0}: Error finding container 943a70a918926cf3a0697d69cc965e4206d5faf6ded18ba2bea4132005e03652: Status 404 returned error can't find the container with id 943a70a918926cf3a0697d69cc965e4206d5faf6ded18ba2bea4132005e03652
Sep 19 09:25:05.202281 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.202250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" event={"ID":"bfbe2120-7216-42cc-bc62-421846081679","Type":"ContainerStarted","Data":"943a70a918926cf3a0697d69cc965e4206d5faf6ded18ba2bea4132005e03652"}
Sep 19 09:25:05.399690 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:05.399657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"]
Sep 19 09:25:05.402539 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:25:05.402508    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf799ce05_e5d6_42e4_b145_0a4a534f84b8.slice/crio-056090ccba0fa6f1a9c617a9c5153a4ef4d898d2ed8991f4b14f239af6bcf77c WatchSource:0}: Error finding container 056090ccba0fa6f1a9c617a9c5153a4ef4d898d2ed8991f4b14f239af6bcf77c: Status 404 returned error can't find the container with id 056090ccba0fa6f1a9c617a9c5153a4ef4d898d2ed8991f4b14f239af6bcf77c
Sep 19 09:25:06.207466 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.207423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" event={"ID":"f799ce05-e5d6-42e4-b145-0a4a534f84b8","Type":"ContainerStarted","Data":"578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd"}
Sep 19 09:25:06.207979 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.207484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:06.207979 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.207508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" event={"ID":"f799ce05-e5d6-42e4-b145-0a4a534f84b8","Type":"ContainerStarted","Data":"056090ccba0fa6f1a9c617a9c5153a4ef4d898d2ed8991f4b14f239af6bcf77c"}
Sep 19 09:25:06.208909 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.208883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" event={"ID":"bfbe2120-7216-42cc-bc62-421846081679","Type":"ContainerStarted","Data":"b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"}
Sep 19 09:25:06.226117 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.226071    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" podStartSLOduration=2.226058716 podStartE2EDuration="2.226058716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:25:06.225200488 +0000 UTC m=+3858.411392473" watchObservedRunningTime="2026-09-19 09:25:06.226058716 +0000 UTC m=+3858.412250672"
Sep 19 09:25:06.242619 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.242574    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" podStartSLOduration=2.24256079 podStartE2EDuration="2.24256079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:25:06.24067344 +0000 UTC m=+3858.426865393" watchObservedRunningTime="2026-09-19 09:25:06.24256079 +0000 UTC m=+3858.428752793"
Sep 19 09:25:06.895742 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:06.895701    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:14.895699 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:14.895662    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:14.908128 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:14.908099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:15.259129 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:15.259091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:25:37.217515 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:37.217481    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:25:59.779976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:59.779929    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"]
Sep 19 09:25:59.782930 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:59.782892    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"]
Sep 19 09:25:59.789810 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:25:59.789767    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 09:25:59.789810 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:25:59.789810 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 09:25:59.789810 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9" cmd=["/bin/sleep","15"]
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:59.789823    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" message="" err=<
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:25:59.789834    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" podUID="bfbe2120-7216-42cc-bc62-421846081679" containerName="main" containerID="cri-o://b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"
Sep 19 09:25:59.790098 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:25:59.789876    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" podUID="bfbe2120-7216-42cc-bc62-421846081679" containerName="main" containerID="cri-o://b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9" gracePeriod=60
Sep 19 09:26:00.112137 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.112099    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"]
Sep 19 09:26:00.162189 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.162164    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:26:00.248664 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.248631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9zw2\" (UniqueName: \"kubernetes.io/projected/bfbe2120-7216-42cc-bc62-421846081679-kube-api-access-v9zw2\") pod \"bfbe2120-7216-42cc-bc62-421846081679\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") "
Sep 19 09:26:00.248865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.248686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbe2120-7216-42cc-bc62-421846081679-tls-certs\") pod \"bfbe2120-7216-42cc-bc62-421846081679\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") "
Sep 19 09:26:00.248865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.248709    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-dshm\") pod \"bfbe2120-7216-42cc-bc62-421846081679\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") "
Sep 19 09:26:00.248865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.248725    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-tmp-dir\") pod \"bfbe2120-7216-42cc-bc62-421846081679\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") "
Sep 19 09:26:00.248865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.248756    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-model-cache\") pod \"bfbe2120-7216-42cc-bc62-421846081679\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") "
Sep 19 09:26:00.248865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.248787    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-home\") pod \"bfbe2120-7216-42cc-bc62-421846081679\" (UID: \"bfbe2120-7216-42cc-bc62-421846081679\") "
Sep 19 09:26:00.249165 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.249004    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bfbe2120-7216-42cc-bc62-421846081679" (UID: "bfbe2120-7216-42cc-bc62-421846081679"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:26:00.249165 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.249035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-model-cache" (OuterVolumeSpecName: "model-cache") pod "bfbe2120-7216-42cc-bc62-421846081679" (UID: "bfbe2120-7216-42cc-bc62-421846081679"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:26:00.249165 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.249121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-home" (OuterVolumeSpecName: "home") pod "bfbe2120-7216-42cc-bc62-421846081679" (UID: "bfbe2120-7216-42cc-bc62-421846081679"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:26:00.251081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.251057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbe2120-7216-42cc-bc62-421846081679-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfbe2120-7216-42cc-bc62-421846081679" (UID: "bfbe2120-7216-42cc-bc62-421846081679"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:26:00.251188 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.251091    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-dshm" (OuterVolumeSpecName: "dshm") pod "bfbe2120-7216-42cc-bc62-421846081679" (UID: "bfbe2120-7216-42cc-bc62-421846081679"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:26:00.251188 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.251102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbe2120-7216-42cc-bc62-421846081679-kube-api-access-v9zw2" (OuterVolumeSpecName: "kube-api-access-v9zw2") pod "bfbe2120-7216-42cc-bc62-421846081679" (UID: "bfbe2120-7216-42cc-bc62-421846081679"). InnerVolumeSpecName "kube-api-access-v9zw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:26:00.349913 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.349835    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfbe2120-7216-42cc-bc62-421846081679-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:00.349913 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.349861    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:00.349913 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.349887    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:00.349913 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.349895    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:00.349913 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.349902    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfbe2120-7216-42cc-bc62-421846081679-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:00.350186 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.349920    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zw2\" (UniqueName: \"kubernetes.io/projected/bfbe2120-7216-42cc-bc62-421846081679-kube-api-access-v9zw2\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:00.447897 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.447863    2580 generic.go:356] "Generic (PLEG): container finished" podID="bfbe2120-7216-42cc-bc62-421846081679" containerID="b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9" exitCode=0
Sep 19 09:26:00.448081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.447917    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"
Sep 19 09:26:00.448081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.447953    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" event={"ID":"bfbe2120-7216-42cc-bc62-421846081679","Type":"ContainerDied","Data":"b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"}
Sep 19 09:26:00.448081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.448031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj" event={"ID":"bfbe2120-7216-42cc-bc62-421846081679","Type":"ContainerDied","Data":"943a70a918926cf3a0697d69cc965e4206d5faf6ded18ba2bea4132005e03652"}
Sep 19 09:26:00.448081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.448054    2580 scope.go:117] "RemoveContainer" containerID="b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"
Sep 19 09:26:00.456809 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.456793    2580 scope.go:117] "RemoveContainer" containerID="b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"
Sep 19 09:26:00.457066 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:26:00.457048    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9\": container with ID starting with b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9 not found: ID does not exist" containerID="b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"
Sep 19 09:26:00.457115 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.457075    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9"} err="failed to get container status \"b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9\": rpc error: code = NotFound desc = could not find container \"b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9\": container with ID starting with b481312a89a301506cd1f23e3f851a7143826efa5230d582da390d77a4ec13a9 not found: ID does not exist"
Sep 19 09:26:00.471657 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.471633    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"]
Sep 19 09:26:00.476114 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:00.476094    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-dsqbj"]
Sep 19 09:26:02.078987 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:02.078949    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbe2120-7216-42cc-bc62-421846081679" path="/var/lib/kubelet/pods/bfbe2120-7216-42cc-bc62-421846081679/volumes"
Sep 19 09:26:06.823813 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:06.823770    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:07.217607 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:07.217569    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:14.802360 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:14.802296    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main" containerID="cri-o://578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd" gracePeriod=45
Sep 19 09:26:15.131630 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:15.131513    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main" containerID="cri-o://86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107" gracePeriod=45
Sep 19 09:26:16.824309 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:16.824264    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:17.217804 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:17.217761    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:26.823738 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:26.823690    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:26.824150 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:26.823797    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:26:27.217458 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:27.217420    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:27.217659 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:27.217525    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:26:36.824069 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:36.823976    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:37.218044 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:37.218006    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:26:45.047869 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.047839    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:26:45.226818 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.226781    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f799ce05-e5d6-42e4-b145-0a4a534f84b8-tls-certs\") pod \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") "
Sep 19 09:26:45.226999 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.226924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqdh9\" (UniqueName: \"kubernetes.io/projected/f799ce05-e5d6-42e4-b145-0a4a534f84b8-kube-api-access-bqdh9\") pod \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\" (UID: \"f799ce05-e5d6-42e4-b145-0a4a534f84b8\") "
Sep 19 09:26:45.229081 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.229053    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f799ce05-e5d6-42e4-b145-0a4a534f84b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f799ce05-e5d6-42e4-b145-0a4a534f84b8" (UID: "f799ce05-e5d6-42e4-b145-0a4a534f84b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:26:45.229238 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.229166    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f799ce05-e5d6-42e4-b145-0a4a534f84b8-kube-api-access-bqdh9" (OuterVolumeSpecName: "kube-api-access-bqdh9") pod "f799ce05-e5d6-42e4-b145-0a4a534f84b8" (UID: "f799ce05-e5d6-42e4-b145-0a4a534f84b8"). InnerVolumeSpecName "kube-api-access-bqdh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:26:45.327750 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.327718    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqdh9\" (UniqueName: \"kubernetes.io/projected/f799ce05-e5d6-42e4-b145-0a4a534f84b8-kube-api-access-bqdh9\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:45.327750 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.327747    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f799ce05-e5d6-42e4-b145-0a4a534f84b8-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:45.361057 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.361026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:26:45.529057 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.529029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3bed6b-3c89-4175-85ed-91150def9f8a-tls-certs\") pod \"0d3bed6b-3c89-4175-85ed-91150def9f8a\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") "
Sep 19 09:26:45.529282 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.529108    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f5fg\" (UniqueName: \"kubernetes.io/projected/0d3bed6b-3c89-4175-85ed-91150def9f8a-kube-api-access-7f5fg\") pod \"0d3bed6b-3c89-4175-85ed-91150def9f8a\" (UID: \"0d3bed6b-3c89-4175-85ed-91150def9f8a\") "
Sep 19 09:26:45.531282 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.531250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3bed6b-3c89-4175-85ed-91150def9f8a-kube-api-access-7f5fg" (OuterVolumeSpecName: "kube-api-access-7f5fg") pod "0d3bed6b-3c89-4175-85ed-91150def9f8a" (UID: "0d3bed6b-3c89-4175-85ed-91150def9f8a"). InnerVolumeSpecName "kube-api-access-7f5fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:26:45.531282 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.531277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d3bed6b-3c89-4175-85ed-91150def9f8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d3bed6b-3c89-4175-85ed-91150def9f8a" (UID: "0d3bed6b-3c89-4175-85ed-91150def9f8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:26:45.630556 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.630526    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d3bed6b-3c89-4175-85ed-91150def9f8a-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:45.630556 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.630554    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f5fg\" (UniqueName: \"kubernetes.io/projected/0d3bed6b-3c89-4175-85ed-91150def9f8a-kube-api-access-7f5fg\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:26:45.642332 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.642302    2580 generic.go:356] "Generic (PLEG): container finished" podID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerID="578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd" exitCode=0
Sep 19 09:26:45.642458 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.642349    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"
Sep 19 09:26:45.642458 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.642377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" event={"ID":"f799ce05-e5d6-42e4-b145-0a4a534f84b8","Type":"ContainerDied","Data":"578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd"}
Sep 19 09:26:45.642458 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.642418    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22" event={"ID":"f799ce05-e5d6-42e4-b145-0a4a534f84b8","Type":"ContainerDied","Data":"056090ccba0fa6f1a9c617a9c5153a4ef4d898d2ed8991f4b14f239af6bcf77c"}
Sep 19 09:26:45.642458 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.642447    2580 scope.go:117] "RemoveContainer" containerID="578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd"
Sep 19 09:26:45.643744 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.643720    2580 generic.go:356] "Generic (PLEG): container finished" podID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerID="86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107" exitCode=0
Sep 19 09:26:45.643823 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.643768    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"
Sep 19 09:26:45.643823 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.643802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" event={"ID":"0d3bed6b-3c89-4175-85ed-91150def9f8a","Type":"ContainerDied","Data":"86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107"}
Sep 19 09:26:45.643926 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.643859    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc" event={"ID":"0d3bed6b-3c89-4175-85ed-91150def9f8a","Type":"ContainerDied","Data":"d0821a8d62ed994f1e852b3e79a0337c46ed13434d8350c3e21452f7032e2ade"}
Sep 19 09:26:45.650817 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.650789    2580 scope.go:117] "RemoveContainer" containerID="578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd"
Sep 19 09:26:45.651080 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:26:45.651059    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd\": container with ID starting with 578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd not found: ID does not exist" containerID="578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd"
Sep 19 09:26:45.651160 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.651093    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd"} err="failed to get container status \"578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd\": rpc error: code = NotFound desc = could not find container \"578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd\": container with ID starting with 578f416641d5dbc02f446e8dd89d9950501713999e84fb3405345327a07479cd not found: ID does not exist"
Sep 19 09:26:45.651160 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.651117    2580 scope.go:117] "RemoveContainer" containerID="86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107"
Sep 19 09:26:45.659658 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.659642    2580 scope.go:117] "RemoveContainer" containerID="86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107"
Sep 19 09:26:45.659941 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:26:45.659923    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107\": container with ID starting with 86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107 not found: ID does not exist" containerID="86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107"
Sep 19 09:26:45.660004 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.659946    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107"} err="failed to get container status \"86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107\": rpc error: code = NotFound desc = could not find container \"86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107\": container with ID starting with 86685299bae7093b67b59085c10754d53f919936dac3a3f76c7cf1d9ee91a107 not found: ID does not exist"
Sep 19 09:26:45.668726 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.668705    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"]
Sep 19 09:26:45.673592 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.673571    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbsst22"]
Sep 19 09:26:45.683812 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.683792    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"]
Sep 19 09:26:45.688557 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:45.688537    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-wb2kc"]
Sep 19 09:26:46.078916 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:46.078833    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" path="/var/lib/kubelet/pods/0d3bed6b-3c89-4175-85ed-91150def9f8a/volumes"
Sep 19 09:26:46.079307 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:26:46.079183    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" path="/var/lib/kubelet/pods/f799ce05-e5d6-42e4-b145-0a4a534f84b8/volumes"
Sep 19 09:27:11.397599 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397560    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"]
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397851    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397862    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397879    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397885    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397902    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbe2120-7216-42cc-bc62-421846081679" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397907    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbe2120-7216-42cc-bc62-421846081679" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397956    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f799ce05-e5d6-42e4-b145-0a4a534f84b8" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397963    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3bed6b-3c89-4175-85ed-91150def9f8a" containerName="main"
Sep 19 09:27:11.398126 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.397970    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbe2120-7216-42cc-bc62-421846081679" containerName="main"
Sep 19 09:27:11.400782 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.400764    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.404384 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.404361    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 19 09:27:11.404779 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.404596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 19 09:27:11.404779 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.404636    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-v7ddh\""
Sep 19 09:27:11.404779 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.404641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 19 09:27:11.404779 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.404669    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-zk87s\""
Sep 19 09:27:11.412106 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.412083    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"]
Sep 19 09:27:11.429174 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.429150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5zn\" (UniqueName: \"kubernetes.io/projected/265af64d-cebe-4ee6-94b2-1b56f24d9269-kube-api-access-mc5zn\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.429336 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.429215    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265af64d-cebe-4ee6-94b2-1b56f24d9269-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.530337 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.530305    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265af64d-cebe-4ee6-94b2-1b56f24d9269-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.530495 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.530351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5zn\" (UniqueName: \"kubernetes.io/projected/265af64d-cebe-4ee6-94b2-1b56f24d9269-kube-api-access-mc5zn\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.532746 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.532723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265af64d-cebe-4ee6-94b2-1b56f24d9269-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.542402 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.542378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5zn\" (UniqueName: \"kubernetes.io/projected/265af64d-cebe-4ee6-94b2-1b56f24d9269-kube-api-access-mc5zn\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:11.710836 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:11.710800    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:12.047010 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:12.046869    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"]
Sep 19 09:27:12.059415 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:27:12.059382    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265af64d_cebe_4ee6_94b2_1b56f24d9269.slice/crio-d1d1aa05a362061c6b51692544d771a0ea927c93fbec614c3251fc2b6144d9c4 WatchSource:0}: Error finding container d1d1aa05a362061c6b51692544d771a0ea927c93fbec614c3251fc2b6144d9c4: Status 404 returned error can't find the container with id d1d1aa05a362061c6b51692544d771a0ea927c93fbec614c3251fc2b6144d9c4
Sep 19 09:27:12.755770 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:12.755730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" event={"ID":"265af64d-cebe-4ee6-94b2-1b56f24d9269","Type":"ContainerStarted","Data":"b9c6bc665e0b85ceeca4fe8a2d30c21221ddffd09a8e2381146a229ba52dcfca"}
Sep 19 09:27:12.756300 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:12.755794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" event={"ID":"265af64d-cebe-4ee6-94b2-1b56f24d9269","Type":"ContainerStarted","Data":"d1d1aa05a362061c6b51692544d771a0ea927c93fbec614c3251fc2b6144d9c4"}
Sep 19 09:27:12.756300 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:12.755837    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:12.779268 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:12.779195    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" podStartSLOduration=1.779180455 podStartE2EDuration="1.779180455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:27:12.777037736 +0000 UTC m=+3984.963229692" watchObservedRunningTime="2026-09-19 09:27:12.779180455 +0000 UTC m=+3984.965372413"
Sep 19 09:27:43.762196 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:43.762164    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:27:45.993514 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:45.993476    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"]
Sep 19 09:27:45.996574 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:45.996556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.006445 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.006416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"]
Sep 19 09:27:46.120332 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.120300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.120516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.120339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpsx\" (UniqueName: \"kubernetes.io/projected/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-kube-api-access-qxpsx\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.120516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.120385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.120516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.120446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.120516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.120484    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.120516 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.120501    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.221501 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.221761 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.221925 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221772    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.221925 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.221925 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.221925 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpsx\" (UniqueName: \"kubernetes.io/projected/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-kube-api-access-qxpsx\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.222140 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.221956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.222140 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.222029    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.222288 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.222174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.224110 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.224091    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.224550 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.224529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.230288 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.230266    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpsx\" (UniqueName: \"kubernetes.io/projected/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-kube-api-access-qxpsx\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-xvrw6\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.308639 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.308577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:46.579398 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.579364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"]
Sep 19 09:27:46.582162 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:27:46.582134    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acb9ba7_e87b_4d3e_9196_f4bea6e9cd3e.slice/crio-1590bd8531f61a0f19d51650d31ede0cf586664379cfe152be0a876e8897f660 WatchSource:0}: Error finding container 1590bd8531f61a0f19d51650d31ede0cf586664379cfe152be0a876e8897f660: Status 404 returned error can't find the container with id 1590bd8531f61a0f19d51650d31ede0cf586664379cfe152be0a876e8897f660
Sep 19 09:27:46.892167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.892061    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" event={"ID":"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e","Type":"ContainerStarted","Data":"5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"}
Sep 19 09:27:46.892167 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.892120    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" event={"ID":"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e","Type":"ContainerStarted","Data":"1590bd8531f61a0f19d51650d31ede0cf586664379cfe152be0a876e8897f660"}
Sep 19 09:27:46.909373 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:46.909325    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" podStartSLOduration=1.909309995 podStartE2EDuration="1.909309995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 09:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:27:46.908117879 +0000 UTC m=+4019.094309836" watchObservedRunningTime="2026-09-19 09:27:46.909309995 +0000 UTC m=+4019.095501952"
Sep 19 09:27:47.309621 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:47.309586    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:54.885317 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:54.885280    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"]
Sep 19 09:27:54.891000 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:54.890971    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"]
Sep 19 09:27:54.894706 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:27:54.894674    2580 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 19 09:27:54.894706 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:27:54.894706 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 09:27:54.894706 ip-10-0-140-191 kubenswrapper[2580]:  > containerID="5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a" cmd=["/bin/sleep","15"]
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:54.894717    2580 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" message="" err=<
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]:  >
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:27:54.894730    2580 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]:         , stdout: , stderr: , exit code -1
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" podUID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" containerName="main" containerID="cri-o://5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"
Sep 19 09:27:54.895007 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:54.894763    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" podUID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" containerName="main" containerID="cri-o://5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a" gracePeriod=60
Sep 19 09:27:55.142503 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.142437    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:55.305386 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305356    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-model-cache\") pod \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") "
Sep 19 09:27:55.305567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305396    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxpsx\" (UniqueName: \"kubernetes.io/projected/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-kube-api-access-qxpsx\") pod \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") "
Sep 19 09:27:55.305567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-home\") pod \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") "
Sep 19 09:27:55.305567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tmp-dir\") pod \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") "
Sep 19 09:27:55.305567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305495    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-dshm\") pod \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") "
Sep 19 09:27:55.305567 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tls-certs\") pod \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\" (UID: \"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e\") "
Sep 19 09:27:55.305978 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305938    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" (UID: "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.305978 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.305958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-home" (OuterVolumeSpecName: "home") pod "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" (UID: "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.306114 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.306026    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" (UID: "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.306203 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.306181    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tmp-dir\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.306286 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.306206    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-model-cache\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.306286 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.306243    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-home\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.313642 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.313583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-dshm" (OuterVolumeSpecName: "dshm") pod "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" (UID: "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 19 09:27:55.313789 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.313702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" (UID: "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:27:55.313923 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.313903    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-kube-api-access-qxpsx" (OuterVolumeSpecName: "kube-api-access-qxpsx") pod "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" (UID: "3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e"). InnerVolumeSpecName "kube-api-access-qxpsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:27:55.407257 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.407161    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-dshm\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.407257 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.407209    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.407257 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.407239    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxpsx\" (UniqueName: \"kubernetes.io/projected/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e-kube-api-access-qxpsx\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:27:55.932536 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.932495    2580 generic.go:356] "Generic (PLEG): container finished" podID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" containerID="5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a" exitCode=0
Sep 19 09:27:55.932976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.932561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"
Sep 19 09:27:55.932976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.932581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" event={"ID":"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e","Type":"ContainerDied","Data":"5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"}
Sep 19 09:27:55.932976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.932670    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6" event={"ID":"3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e","Type":"ContainerDied","Data":"1590bd8531f61a0f19d51650d31ede0cf586664379cfe152be0a876e8897f660"}
Sep 19 09:27:55.932976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.932696    2580 scope.go:117] "RemoveContainer" containerID="5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"
Sep 19 09:27:55.941664 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.941646    2580 scope.go:117] "RemoveContainer" containerID="5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"
Sep 19 09:27:55.941946 ip-10-0-140-191 kubenswrapper[2580]: E0919 09:27:55.941929    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a\": container with ID starting with 5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a not found: ID does not exist" containerID="5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"
Sep 19 09:27:55.941993 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.941954    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a"} err="failed to get container status \"5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a\": rpc error: code = NotFound desc = could not find container \"5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a\": container with ID starting with 5892c9fc3ebe9329153c6d791ddece4d534e0911d048fd659ce3424e654dae9a not found: ID does not exist"
Sep 19 09:27:55.956704 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.956677    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"]
Sep 19 09:27:55.959938 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:55.959917    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-xvrw6"]
Sep 19 09:27:56.078987 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:27:56.078956    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" path="/var/lib/kubelet/pods/3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e/volumes"
Sep 19 09:28:03.761362 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:03.761266    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:28:09.910716 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:09.910653    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main" containerID="cri-o://b9c6bc665e0b85ceeca4fe8a2d30c21221ddffd09a8e2381146a229ba52dcfca" gracePeriod=45
Sep 19 09:28:13.761895 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:13.761856    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:28:23.761907 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:23.761868    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:28:23.762448 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:23.761983    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:28:33.761450 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:33.761408    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 19 09:28:40.117748 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.117715    2580 generic.go:356] "Generic (PLEG): container finished" podID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerID="b9c6bc665e0b85ceeca4fe8a2d30c21221ddffd09a8e2381146a229ba52dcfca" exitCode=0
Sep 19 09:28:40.118135 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.117780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" event={"ID":"265af64d-cebe-4ee6-94b2-1b56f24d9269","Type":"ContainerDied","Data":"b9c6bc665e0b85ceeca4fe8a2d30c21221ddffd09a8e2381146a229ba52dcfca"}
Sep 19 09:28:40.165989 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.165957    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:28:40.260951 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.260907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265af64d-cebe-4ee6-94b2-1b56f24d9269-tls-certs\") pod \"265af64d-cebe-4ee6-94b2-1b56f24d9269\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") "
Sep 19 09:28:40.261165 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.260977    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc5zn\" (UniqueName: \"kubernetes.io/projected/265af64d-cebe-4ee6-94b2-1b56f24d9269-kube-api-access-mc5zn\") pod \"265af64d-cebe-4ee6-94b2-1b56f24d9269\" (UID: \"265af64d-cebe-4ee6-94b2-1b56f24d9269\") "
Sep 19 09:28:40.263240 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.263175    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265af64d-cebe-4ee6-94b2-1b56f24d9269-kube-api-access-mc5zn" (OuterVolumeSpecName: "kube-api-access-mc5zn") pod "265af64d-cebe-4ee6-94b2-1b56f24d9269" (UID: "265af64d-cebe-4ee6-94b2-1b56f24d9269"). InnerVolumeSpecName "kube-api-access-mc5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 19 09:28:40.263240 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.263184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/265af64d-cebe-4ee6-94b2-1b56f24d9269-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "265af64d-cebe-4ee6-94b2-1b56f24d9269" (UID: "265af64d-cebe-4ee6-94b2-1b56f24d9269"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 19 09:28:40.362532 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.362485    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5zn\" (UniqueName: \"kubernetes.io/projected/265af64d-cebe-4ee6-94b2-1b56f24d9269-kube-api-access-mc5zn\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:28:40.362532 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:40.362524    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/265af64d-cebe-4ee6-94b2-1b56f24d9269-tls-certs\") on node \"ip-10-0-140-191.ec2.internal\" DevicePath \"\""
Sep 19 09:28:41.122965 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:41.122937    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"
Sep 19 09:28:41.123427 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:41.122934    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w" event={"ID":"265af64d-cebe-4ee6-94b2-1b56f24d9269","Type":"ContainerDied","Data":"d1d1aa05a362061c6b51692544d771a0ea927c93fbec614c3251fc2b6144d9c4"}
Sep 19 09:28:41.123427 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:41.123068    2580 scope.go:117] "RemoveContainer" containerID="b9c6bc665e0b85ceeca4fe8a2d30c21221ddffd09a8e2381146a229ba52dcfca"
Sep 19 09:28:41.145209 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:41.145182    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"]
Sep 19 09:28:41.147685 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:41.147658    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-hgm4w"]
Sep 19 09:28:42.079571 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:28:42.079539    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" path="/var/lib/kubelet/pods/265af64d-cebe-4ee6-94b2-1b56f24d9269/volumes"
Sep 19 09:29:07.927479 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:07.927444    2580 ???:1] "http: TLS handshake error from 10.0.134.254:60652: EOF"
Sep 19 09:29:08.245733 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:08.245699    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-pcjwk_64a9059d-e7a1-4dad-80d2-392b73ad55b6/authorino/0.log"
Sep 19 09:29:08.288976 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:08.288946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7b6ph_a82c9f71-9816-4398-bc7a-071a0ce1e361/kuadrant-console-plugin/0.log"
Sep 19 09:29:15.230099 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:15.230063    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ttzbh_bdfdbe55-3611-49af-8161-9ba335bbc66f/global-pull-secret-syncer/0.log"
Sep 19 09:29:15.351359 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:15.351321    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m87q2_1c9614d0-96cb-451d-b53e-c442ef9aebb6/konnectivity-agent/0.log"
Sep 19 09:29:15.464189 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:15.464150    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-191.ec2.internal_b28b83e27db47df5e96b9958be391357/haproxy/0.log"
Sep 19 09:29:20.136606 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:20.136561    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-pcjwk_64a9059d-e7a1-4dad-80d2-392b73ad55b6/authorino/0.log"
Sep 19 09:29:20.240400 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:20.240372    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-7b6ph_a82c9f71-9816-4398-bc7a-071a0ce1e361/kuadrant-console-plugin/0.log"
Sep 19 09:29:21.975728 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:21.975696    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-dqcj4_3fcbb230-b4c8-4833-978c-d493b5a29cc0/monitoring-plugin/0.log"
Sep 19 09:29:22.012518 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.012487    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7km6n_eb05e8b9-c157-49e2-b5df-ae7ab0bb6713/node-exporter/0.log"
Sep 19 09:29:22.030828 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.030798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7km6n_eb05e8b9-c157-49e2-b5df-ae7ab0bb6713/kube-rbac-proxy/0.log"
Sep 19 09:29:22.053792 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.053768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7km6n_eb05e8b9-c157-49e2-b5df-ae7ab0bb6713/init-textfile/0.log"
Sep 19 09:29:22.466332 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.466296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-h2kt4_651eab1b-6453-4950-8ffe-90f211f8beae/prometheus-operator/0.log"
Sep 19 09:29:22.491380 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.491346    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-h2kt4_651eab1b-6453-4950-8ffe-90f211f8beae/kube-rbac-proxy/0.log"
Sep 19 09:29:22.523206 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.523179    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-9sxj6_7189d561-869e-46ad-b4c8-a14a08ed144b/prometheus-operator-admission-webhook/0.log"
Sep 19 09:29:22.571642 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.571608    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c786d67c4-w4blg_5920b672-278d-47ef-aff8-8fa99f507301/telemeter-client/0.log"
Sep 19 09:29:22.602443 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.602401    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c786d67c4-w4blg_5920b672-278d-47ef-aff8-8fa99f507301/reload/0.log"
Sep 19 09:29:22.625095 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.625062    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c786d67c4-w4blg_5920b672-278d-47ef-aff8-8fa99f507301/kube-rbac-proxy/0.log"
Sep 19 09:29:22.655283 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.655252    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cfbcf76-qdwb4_261fe2e9-ba35-49c2-8b64-4d15ad40f71f/thanos-query/0.log"
Sep 19 09:29:22.676996 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.676965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cfbcf76-qdwb4_261fe2e9-ba35-49c2-8b64-4d15ad40f71f/kube-rbac-proxy-web/0.log"
Sep 19 09:29:22.696840 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.696810    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cfbcf76-qdwb4_261fe2e9-ba35-49c2-8b64-4d15ad40f71f/kube-rbac-proxy/0.log"
Sep 19 09:29:22.720872 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.720798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cfbcf76-qdwb4_261fe2e9-ba35-49c2-8b64-4d15ad40f71f/prom-label-proxy/0.log"
Sep 19 09:29:22.741802 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.741768    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cfbcf76-qdwb4_261fe2e9-ba35-49c2-8b64-4d15ad40f71f/kube-rbac-proxy-rules/0.log"
Sep 19 09:29:22.763865 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:22.763843    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848cfbcf76-qdwb4_261fe2e9-ba35-49c2-8b64-4d15ad40f71f/kube-rbac-proxy-metrics/0.log"
Sep 19 09:29:24.974053 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:24.974025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8697984585-5vbdz_2230d5ae-7b76-43c2-8bb0-cf989687c6f7/console/0.log"
Sep 19 09:29:25.018885 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.018857    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-5lvxg_d37eff43-5018-490b-a02e-74a9156d3e11/download-server/0.log"
Sep 19 09:29:25.232287 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232138    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"]
Sep 19 09:29:25.232576 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232561    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main"
Sep 19 09:29:25.232631 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232577    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main"
Sep 19 09:29:25.232631 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232585    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" containerName="main"
Sep 19 09:29:25.232631 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232591    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" containerName="main"
Sep 19 09:29:25.232736 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232652    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3acb9ba7-e87b-4d3e-9196-f4bea6e9cd3e" containerName="main"
Sep 19 09:29:25.232736 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.232659    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="265af64d-cebe-4ee6-94b2-1b56f24d9269" containerName="main"
Sep 19 09:29:25.235720 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.235700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.239838 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.239821    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h99lz\"/\"openshift-service-ca.crt\""
Sep 19 09:29:25.239984 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.239959    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h99lz\"/\"default-dockercfg-zpdhd\""
Sep 19 09:29:25.240065 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.240043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h99lz\"/\"kube-root-ca.crt\""
Sep 19 09:29:25.246740 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.246716    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"]
Sep 19 09:29:25.336495 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.336457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfp4\" (UniqueName: \"kubernetes.io/projected/edf7707a-7301-40dc-9f84-9addaa4248e4-kube-api-access-9hfp4\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.336680 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.336507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-lib-modules\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.336680 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.336531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-proc\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.336680 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.336569    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-podres\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.336680 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.336619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-sys\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.437991 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.437952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfp4\" (UniqueName: \"kubernetes.io/projected/edf7707a-7301-40dc-9f84-9addaa4248e4-kube-api-access-9hfp4\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-lib-modules\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438029    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-proc\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-podres\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-sys\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-sys\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438392 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-proc\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438392 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-lib-modules\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.438392 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.438216    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/edf7707a-7301-40dc-9f84-9addaa4248e4-podres\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.446489 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.446467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfp4\" (UniqueName: \"kubernetes.io/projected/edf7707a-7301-40dc-9f84-9addaa4248e4-kube-api-access-9hfp4\") pod \"perf-node-gather-daemonset-sx4kv\" (UID: \"edf7707a-7301-40dc-9f84-9addaa4248e4\") " pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.546176 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.546073    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:25.813542 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:25.813457    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"]
Sep 19 09:29:25.816624 ip-10-0-140-191 kubenswrapper[2580]: W0919 09:29:25.816591    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podedf7707a_7301_40dc_9f84_9addaa4248e4.slice/crio-8eb8718447add1d2b328ae7e07ba9fe0c5a0b6da0960e178f02a546cb5cf9d56 WatchSource:0}: Error finding container 8eb8718447add1d2b328ae7e07ba9fe0c5a0b6da0960e178f02a546cb5cf9d56: Status 404 returned error can't find the container with id 8eb8718447add1d2b328ae7e07ba9fe0c5a0b6da0960e178f02a546cb5cf9d56
Sep 19 09:29:26.310198 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.310163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv" event={"ID":"edf7707a-7301-40dc-9f84-9addaa4248e4","Type":"ContainerStarted","Data":"6b778dd4cb7cdb4193510cf3c153e9efe5013200cd75ead9c5d493b5c627b075"}
Sep 19 09:29:26.310613 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.310215    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv" event={"ID":"edf7707a-7301-40dc-9f84-9addaa4248e4","Type":"ContainerStarted","Data":"8eb8718447add1d2b328ae7e07ba9fe0c5a0b6da0960e178f02a546cb5cf9d56"}
Sep 19 09:29:26.310613 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.310255    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:26.327595 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.327546    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv" podStartSLOduration=1.327531043 podStartE2EDuration="1.327531043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:29:26.326276913 +0000 UTC m=+4118.512468867" watchObservedRunningTime="2026-09-19 09:29:26.327531043 +0000 UTC m=+4118.513723000"
Sep 19 09:29:26.550135 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.550096    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzt4t_19b17a34-96c9-4497-82d3-e359db7fcf43/dns/0.log"
Sep 19 09:29:26.569279 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.569187    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jzt4t_19b17a34-96c9-4497-82d3-e359db7fcf43/kube-rbac-proxy/0.log"
Sep 19 09:29:26.727476 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:26.727451    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q55gs_671d0b83-2a3c-461f-8214-d25f5fbcd63b/dns-node-resolver/0.log"
Sep 19 09:29:27.215492 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:27.215459    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-b48bc997b-sr52z_77669641-85e6-4d4f-a43e-c0082f04e0de/registry/0.log"
Sep 19 09:29:27.288314 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:27.288278    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mpps7_6397ebc5-f249-4d2d-bf1e-183985254161/node-ca/0.log"
Sep 19 09:29:28.822515 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:28.822477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wqgl6_74f1437b-61c4-4a11-bbea-988d75a1d579/serve-healthcheck-canary/0.log"
Sep 19 09:29:29.287699 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:29.287660    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xst4_195ae344-bf5f-4194-9dd0-d36dc7e712ad/kube-rbac-proxy/0.log"
Sep 19 09:29:29.308502 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:29.308469    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xst4_195ae344-bf5f-4194-9dd0-d36dc7e712ad/exporter/0.log"
Sep 19 09:29:29.329348 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:29.329323    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4xst4_195ae344-bf5f-4194-9dd0-d36dc7e712ad/extractor/0.log"
Sep 19 09:29:32.325319 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:32.325239    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h99lz/perf-node-gather-daemonset-sx4kv"
Sep 19 09:29:32.464765 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:32.464732    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6474f95f46-76mq2_a70be341-8383-4f67-a66a-7664ba2fdde0/manager/0.log"
Sep 19 09:29:34.340994 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:34.340959    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-kbp7n_d4a9eecf-8e1b-441f-9196-33028600a189/seaweedfs/0.log"
Sep 19 09:29:40.767972 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.767945    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/kube-multus-additional-cni-plugins/0.log"
Sep 19 09:29:40.787455 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.787430    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/egress-router-binary-copy/0.log"
Sep 19 09:29:40.805963 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.805938    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/cni-plugins/0.log"
Sep 19 09:29:40.826575 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.826555    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/bond-cni-plugin/0.log"
Sep 19 09:29:40.845249 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.845210    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/routeoverride-cni/0.log"
Sep 19 09:29:40.865756 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.865731    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/whereabouts-cni-bincopy/0.log"
Sep 19 09:29:40.884637 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.884616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xtf4w_ab3c3f13-22e3-43af-ad98-06fb882fd52b/whereabouts-cni/0.log"
Sep 19 09:29:40.980435 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:40.980401    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-whtmv_56d022e4-98aa-4524-9f4a-cd7098b35158/kube-multus/0.log"
Sep 19 09:29:41.080946 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:41.080853    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dkxrb_182dd4b2-6f32-484f-9fcc-7bd0de43f465/network-metrics-daemon/0.log"
Sep 19 09:29:41.095206 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:41.095180    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dkxrb_182dd4b2-6f32-484f-9fcc-7bd0de43f465/kube-rbac-proxy/0.log"
Sep 19 09:29:42.189445 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.189409    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/ovn-controller/0.log"
Sep 19 09:29:42.204709 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.204683    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/ovn-acl-logging/0.log"
Sep 19 09:29:42.226304 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.226273    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/kube-rbac-proxy-node/0.log"
Sep 19 09:29:42.245964 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.245927    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/kube-rbac-proxy-ovn-metrics/0.log"
Sep 19 09:29:42.264192 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.264165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/northd/0.log"
Sep 19 09:29:42.285190 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.285169    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/nbdb/0.log"
Sep 19 09:29:42.304084 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.304058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/sbdb/0.log"
Sep 19 09:29:42.394825 ip-10-0-140-191 kubenswrapper[2580]: I0919 09:29:42.394793    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-78n4r_feb84e54-ff61-45af-9c19-a3b231e5c3e0/ovnkube-controller/0.log"