Sep 16 17:22:54.446576 ip-10-0-128-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 17:22:54.446586 ip-10-0-128-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 17:22:54.446596 ip-10-0-128-156 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 17:22:54.446765 ip-10-0-128-156 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 17:23:04.487990 ip-10-0-128-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 17:23:04.488009 ip-10-0-128-156 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 534de5b4a85242869826cc296b4c985b --
Sep 16 17:24:43.393392 ip-10-0-128-156 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 17:24:43.817744 ip-10-0-128-156 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:43.817744 ip-10-0-128-156 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 17:24:43.817744 ip-10-0-128-156 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:43.817744 ip-10-0-128-156 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 17:24:43.817744 ip-10-0-128-156 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 17:24:43.818718 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.818633    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 17:24:43.823951 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823922    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:43.823951 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823944    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:43.823951 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823950    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:43.823951 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823953    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:43.823951 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823956    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:43.823951 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823960    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823963    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823968    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823971    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823974    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823977    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823980    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823982    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823985    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823988    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823990    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823993    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823996    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.823998    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824001    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824003    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824006    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824009    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824017    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:43.824203 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824020    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824023    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824026    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824028    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824032    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824034    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824037    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824040    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824043    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824045    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824047    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824050    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824053    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824055    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824058    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824061    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824064    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824067    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824069    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824072    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:43.824663 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824074    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824077    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824080    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824082    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824085    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824087    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824107    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824110    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824113    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824116    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824119    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824122    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824124    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824127    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824130    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824132    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824135    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824137    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824140    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824143    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:43.825213 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824145    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824148    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824153    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824156    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824159    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824162    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824165    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824169    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824172    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824176    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824180    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824184    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824188    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824192    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824196    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824200    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824205    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824208    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824210    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:43.825792 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824215    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824218    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824221    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824224    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824226    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824229    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824231    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824234    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824236    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824239    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824241    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824244    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824246    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824249    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824252    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824254    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824966    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824974    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824979    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:43.826284 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824983    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824987    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824990    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824993    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824996    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.824999    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825002    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825004    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825007    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825009    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825012    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825014    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825017    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825021    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825026    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825030    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825033    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825036    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825038    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:43.826744 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825041    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825044    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825046    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825049    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825051    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825054    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825057    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825059    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825062    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825064    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825067    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825071    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825075    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825078    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825082    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825085    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825088    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825107    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825111    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:43.827235 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825113    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825116    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825119    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825121    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825124    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825126    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825129    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825132    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825134    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825137    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825140    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825143    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825145    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825148    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825151    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825153    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825156    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825158    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825161    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825165    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:43.827720 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825168    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825171    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825173    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825176    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825178    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825181    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825183    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825186    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825189    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825192    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825194    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825197    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825199    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825202    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825205    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825207    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825210    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825212    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825215    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825218    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:43.828228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825221    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825223    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825226    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825229    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825232    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825234    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825237    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825239    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825242    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825244    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825247    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825249    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825252    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825254    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825257    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825260    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825262    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825265    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825647    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825652    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825655    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:43.828708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825658    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825661    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825663    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825666    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825669    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825672    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825674    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825677    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825680    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825683    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825687    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825690    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825693    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825697    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825700    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825702    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825705    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825707    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825710    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:43.829261 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825712    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825715    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825717    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825720    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825723    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825725    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825728    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825731    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825733    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825737    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825743    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825747    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825750    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825753    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825756    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825759    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825763    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825765    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:43.829726 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825769    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825772    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825774    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825777    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825780    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825782    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825785    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825787    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825790    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825793    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825796    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825799    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825802    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825805    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825807    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825810    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825813    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825815    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825818    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825820    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:43.830194 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825823    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825825    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825828    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825830    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825834    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825838    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825841    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825843    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825846    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825848    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825852    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825854    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825857    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825860    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825862    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825865    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825867    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825870    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825872    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825875    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:43.830713 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825877    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825880    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825883    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825885    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825888    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825891    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825893    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825896    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825898    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825901    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825903    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825906    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825908    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825911    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825913    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825915    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825919    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825923    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.825926    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826006    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826039    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 17:24:43.831256 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826046    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826050    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826054    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826057    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826061    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826066    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826070    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826072    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826076    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826079    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826082    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826085    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826088    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826104    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826108    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826111    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826115    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826118    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826121    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826124    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826128    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826131    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826134    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 17:24:43.831815 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826138    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826141    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826144    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826147    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826152    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826157    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826160    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826164    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826167    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826170    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826173    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826176    2560 flags.go:64] FLAG: --enable-server="true"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826178    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826183    2560 flags.go:64] FLAG: --event-burst="100"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826187    2560 flags.go:64] FLAG: --event-qps="50"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826190    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826193    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826196    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826200    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826203    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826206    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826209    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826212    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826214    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826218    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 17:24:43.832379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826221    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826224    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826227    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826230    2560 flags.go:64] FLAG: --feature-gates=""
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826234    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826237    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826240    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826243    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826246    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826249    2560 flags.go:64] FLAG: --help="false"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826252    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-128-156.ec2.internal"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826256    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826261    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826264    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826267    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826271    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826274    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826276    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826279    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826282    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826285    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826288    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826291    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826294    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 17:24:43.833013 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826297    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826300    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826303    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826305    2560 flags.go:64] FLAG: --lock-file=""
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826308    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826311    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826314    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826320    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826323    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826326    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826329    2560 flags.go:64] FLAG: --logging-format="text"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826332    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826335    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826338    2560 flags.go:64] FLAG: --manifest-url=""
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826341    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826345    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826348    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826351    2560 flags.go:64] FLAG: --max-pods="110"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826354    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826358    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826362    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826364    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826368    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826370    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826373    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 17:24:43.833645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826380    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826383    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826386    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826389    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826392    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826398    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826401    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826404    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826407    2560 flags.go:64] FLAG: --port="10250"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826410    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826413    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0aca9dfc007bd5652"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826416    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826419    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826422    2560 flags.go:64] FLAG: --register-node="true"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826424    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826428    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826431    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826434    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826436    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826440    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826443    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826446    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826448    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826451    2560 flags.go:64] FLAG: --runonce="false"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826454    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 17:24:43.834273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826456    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826460    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826463    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826466    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826469    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826472    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826475    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826478    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826481    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826484    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826486    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826489    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826492    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826495    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826501    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826504    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826506    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826510    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826513    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826516    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826518    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826521    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826524    2560 flags.go:64] FLAG: --v="2"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826528    2560 flags.go:64] FLAG: --version="false"
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826532    2560 flags.go:64] FLAG: --vmodule=""
Sep 16 17:24:43.834882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826536    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.826539    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826649    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826653    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826657    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826659    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826662    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826665    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826668    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826670    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826674    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826676    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826679    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826681    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826684    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826686    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826689    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826692    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826695    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826697    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:43.835539 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826700    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826702    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826705    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826707    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826710    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826713    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826715    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826718    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826720    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826723    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826725    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826728    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826732    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826736    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826739    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826743    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826747    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826750    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826753    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:43.836061 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826755    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826759    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826762    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826765    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826768    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826770    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826773    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826775    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826778    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826780    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826783    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826785    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826788    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826790    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826793    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826795    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826798    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826800    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826803    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826805    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:43.836538 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826808    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826810    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826813    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826816    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826818    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826820    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826823    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826826    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826829    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826831    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826833    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826836    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826839    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826843    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826845    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826851    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826853    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826856    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826858    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826861    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:43.837014 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826864    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826866    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826869    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826872    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826874    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826878    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826880    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826883    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826887    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826892    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826897    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826902    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826905    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826908    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826911    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826914    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826917    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826919    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826922    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:43.837514 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826926    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826929    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.826931    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.827514    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.834524    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.834539    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.835402    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.835410    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835465    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835472    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835475    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835479    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835482    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835485    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835488    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:43.837977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835491    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835494    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835497    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835500    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835503    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835506    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835509    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835512    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835515    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835517    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835520    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835523    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835525    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835528    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835531    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835534    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835536    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835539    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835542    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835545    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:43.838371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835547    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835550    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835554    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835557    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835560    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835563    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835565    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835568    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835570    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835573    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835576    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835578    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835581    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835583    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835586    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835588    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835591    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835593    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835596    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835599    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:43.838895 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835602    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835604    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835607    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835609    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835612    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835614    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835617    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835619    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835622    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835624    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835627    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835630    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835632    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835643    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835646    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835650    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835653    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835655    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835658    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:43.839398 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835661    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835663    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835666    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835668    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835671    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835674    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835676    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835679    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835681    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835684    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835686    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835688    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835691    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835693    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835696    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835698    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835702    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835705    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835709    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:43.839873 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835711    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835714    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835717    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835719    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835722    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835724    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835726    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835729    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835732    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835735    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835738    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835742    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835746    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835748    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.835754    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:43.840348 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835862    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835867    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835870    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835873    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835876    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835878    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835881    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835884    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835887    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835890    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835892    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835895    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835897    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835900    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835903    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835905    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835908    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835911    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835913    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835916    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 17:24:43.840724 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835918    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835921    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835923    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835927    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835931    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835934    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835937    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835940    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835943    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835946    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835949    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835951    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835954    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835957    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835959    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835962    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835964    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835967    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835970    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835972    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 17:24:43.841228 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835975    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835977    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835980    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835982    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835994    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835997    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.835999    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836002    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836005    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836007    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836010    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836012    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836015    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836017    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836020    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836023    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836025    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836029    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836032    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 17:24:43.841708 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836035    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836038    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836047    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836051    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836054    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836057    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836059    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836062    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836065    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836067    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836070    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836073    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836075    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836078    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836080    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836083    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836085    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836088    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836102    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 17:24:43.842298 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836105    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836107    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836110    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836113    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836115    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836117    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836120    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836122    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836125    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836127    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836130    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836133    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836135    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836138    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836141    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836143    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836151    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836154    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836157    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836159    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 17:24:43.842751 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:43.836161    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.836166    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.836266    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.839359    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.840412    2560 server.go:1019] "Starting client certificate rotation"
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.840512    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.840538    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 17:24:43.843248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.843228    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:43.845488 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.845470    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 17:24:43.855500 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.855482    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 16 17:24:43.858756 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.858743    2560 log.go:25] "Validated CRI v1 image API"
Sep 16 17:24:43.860172 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.860154    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 17:24:43.866355 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.866331    2560 fs.go:135] Filesystem UUIDs: map[16272b80-3103-43ee-9fd3-72faa991ab98:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 aa97a8f3-78b7-4505-8b53-674317c5b802:/dev/nvme0n1p4]
Sep 16 17:24:43.866437 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.866353    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 17:24:43.870528 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.870423    2560 manager.go:216] Machine: {Timestamp:2026-09-16 17:24:43.86994437 +0000 UTC m=+0.355556801 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098336 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2396801b291f0db7a7c29bdee5059d SystemUUID:ec239680-1b29-1f0d-b7a7-c29bdee5059d BootID:534de5b4-a852-4286-9826-cc296b4c985b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e2:9b:e5:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e2:9b:e5:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:cc:78:43:33:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 17:24:43.870528 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.870522    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 17:24:43.870629 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.870590    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 17:24:43.873804 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.873778    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 17:24:43.873935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.873806    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 17:24:43.875120 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.875111    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 17:24:43.875153 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.875123    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 17:24:43.875153 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.875132    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:43.876161 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.876150    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 17:24:43.877116 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.877107    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 17:24:43.878746 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.878731    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:43.880409 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880397    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:43.880535 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880526    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 17:24:43.880565 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880555    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 17:24:43.880565 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880564    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 17:24:43.880623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880578    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 17:24:43.880623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880586    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 16 17:24:43.880623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880594    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 17:24:43.880899 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880888    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:43.880933 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.880912    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 17:24:43.883258 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.883241    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 17:24:43.885678 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.885660    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 17:24:43.885780 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.885693    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 17:24:43.887314 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887302    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887318    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887325    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887330    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887339    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887348    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887366    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 17:24:43.887372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887371    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 17:24:43.887560 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887377    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 17:24:43.887560 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887382    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 17:24:43.887560 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887391    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 17:24:43.887560 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.887397    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 17:24:43.888265 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.888255    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 17:24:43.888299 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.888268    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 17:24:43.892694 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.892674    2560 server.go:1268] "Started kubelet"
Sep 16 17:24:43.892788 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.892737    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 17:24:43.892788 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.892771    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 17:24:43.892900 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.892828    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 17:24:43.892900 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.892851    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 17:24:43.893046 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.893021    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 17:24:43.893140 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.892870    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 17:24:43.893515 ip-10-0-128-156 systemd[1]: Started Kubernetes Kubelet.
Sep 16 17:24:43.894279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.894260    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 17:24:43.894740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.894727    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 17:24:43.897678 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.897656    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 17:24:43.900113 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.898952    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1abf87de  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.892647902 +0000 UTC m=+0.378260336,LastTimestamp:2026-09-16 17:24:43.892647902 +0000 UTC m=+0.378260336,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.900190 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900133    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 17:24:43.900190 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900156    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:43.900275 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.900219    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:43.900724 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900708    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 17:24:43.900804 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900734    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 17:24:43.900804 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900747    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 17:24:43.900945 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900911    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 17:24:43.900945 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.900922    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 17:24:43.901222 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901202    2560 factory.go:55] Registering systemd factory
Sep 16 17:24:43.901312 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901232    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 16 17:24:43.901438 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901421    2560 factory.go:153] Registering CRI-O factory
Sep 16 17:24:43.901497 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901442    2560 factory.go:223] Registration of the crio container factory successfully
Sep 16 17:24:43.901497 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901491    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 17:24:43.901561 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901509    2560 factory.go:103] Registering Raw factory
Sep 16 17:24:43.901561 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901525    2560 manager.go:1196] Started watching for new ooms in manager
Sep 16 17:24:43.901970 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.901957    2560 manager.go:349] Starting recovery of all containers
Sep 16 17:24:43.912467 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.912307    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 17:24:43.914778 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.914703    2560 manager.go:354] Recovery completed
Sep 16 17:24:43.916172 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.916144    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 17:24:43.919224 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.919213    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:43.921132 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921086    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:43.921218 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921146    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:43.921218 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921161    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:43.921769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921756    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 17:24:43.921769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921769    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 17:24:43.921890 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921774    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:43.921890 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.921786    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 17:24:43.922505 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.922486    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:43.922585 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.922517    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:43.922585 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.922533    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:43.923109 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.923081    2560 policy_none.go:49] "None policy: Start"
Sep 16 17:24:43.923175 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.923114    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 17:24:43.923175 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.923128    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 17:24:43.924275 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.924261    2560 policy_none.go:47] "Start"
Sep 16 17:24:43.926669 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.926605    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1c7229d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.921131993 +0000 UTC m=+0.406744430,LastTimestamp:2026-09-16 17:24:43.921131993 +0000 UTC m=+0.406744430,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.931483 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.931400    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1c727ca2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.921153186 +0000 UTC m=+0.406765619,LastTimestamp:2026-09-16 17:24:43.921153186 +0000 UTC m=+0.406765619,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.938133 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.938053    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1c72b8cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-156.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.921168588 +0000 UTC m=+0.406781020,LastTimestamp:2026-09-16 17:24:43.921168588 +0000 UTC m=+0.406781020,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.941765 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.941683    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-156.ec2.internal.18d5dd0f1c7229d9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1c7229d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.921131993 +0000 UTC m=+0.406744430,LastTimestamp:2026-09-16 17:24:43.922502673 +0000 UTC m=+0.408115110,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.946582 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.946565    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwx77"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.949365    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-156.ec2.internal.18d5dd0f1c727ca2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1c727ca2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.921153186 +0000 UTC m=+0.406765619,LastTimestamp:2026-09-16 17:24:43.922524237 +0000 UTC m=+0.408136670,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.957420    2560 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-156.ec2.internal.18d5dd0f1c72b8cc\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1c72b8cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-156.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.921168588 +0000 UTC m=+0.406781020,LastTimestamp:2026-09-16 17:24:43.922538932 +0000 UTC m=+0.408151368,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.960726    2560 manager.go:337] "Starting Device Plugin manager"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.960775    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.960788    2560 server.go:87] "Starting device plugin registration server"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.961204    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.961216    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.961307    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.961374    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.961379    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.962166    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.962225    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:43.962975    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwx77"
Sep 16 17:24:43.965983 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:43.965457    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-156.ec2.internal.18d5dd0f1ef00129  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-156.ec2.internal,UID:ip-10-0-128-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-156.ec2.internal,},FirstTimestamp:2026-09-16 17:24:43.962933545 +0000 UTC m=+0.448545963,LastTimestamp:2026-09-16 17:24:43.962933545 +0000 UTC m=+0.448545963,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-156.ec2.internal,}"
Sep 16 17:24:44.033872 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.033835    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 17:24:44.035059 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.035043    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 17:24:44.035137 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.035107    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 17:24:44.035186 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.035141    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 17:24:44.035267 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.035252    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 17:24:44.044716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.044697    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:44.061430 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.061413    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:44.062307 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.062285    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:44.062411 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.062314    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:44.062411 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.062327    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:44.062411 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.062356    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.074389 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.074320    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.074389 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.074342    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-156.ec2.internal\": node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:44.091929 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.091904    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:44.135974 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.135952    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"]
Sep 16 17:24:44.136068 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.136020    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:44.137437 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.137412    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:44.137524 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.137450    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:44.137524 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.137464    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:44.138804 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.138789    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:44.138927 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.138914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.139009 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.138941    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:44.139505 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.139490    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:44.139585 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.139522    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:44.139585 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.139536    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:44.139585 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.139494    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:44.139734 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.139588    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:44.139734 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.139600    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:44.140781 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.140766    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.140861 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.140793    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:44.141491 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.141476    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:44.141559 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.141499    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:44.141559 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.141514    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:44.168621 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.168602    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-156.ec2.internal\" not found" node="ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.172081 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.172067    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-156.ec2.internal\" not found" node="ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.192457 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.192436    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:44.202797 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.202781    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f537d23616ee809add516752b858b10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal\" (UID: \"3f537d23616ee809add516752b858b10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.202847 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.202805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f537d23616ee809add516752b858b10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal\" (UID: \"3f537d23616ee809add516752b858b10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.202847 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.202820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1d152f82d086cc4921e20e44fe174b4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-156.ec2.internal\" (UID: \"e1d152f82d086cc4921e20e44fe174b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.293269 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.293236    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:44.303574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.303557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f537d23616ee809add516752b858b10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal\" (UID: \"3f537d23616ee809add516752b858b10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.303650 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.303602    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f537d23616ee809add516752b858b10-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal\" (UID: \"3f537d23616ee809add516752b858b10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.303682 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.303662    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f537d23616ee809add516752b858b10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal\" (UID: \"3f537d23616ee809add516752b858b10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.303717 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.303694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1d152f82d086cc4921e20e44fe174b4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-156.ec2.internal\" (UID: \"e1d152f82d086cc4921e20e44fe174b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.303750 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.303722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1d152f82d086cc4921e20e44fe174b4-config\") pod \"kube-apiserver-proxy-ip-10-0-128-156.ec2.internal\" (UID: \"e1d152f82d086cc4921e20e44fe174b4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.303750 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:44.303739    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f537d23616ee809add516752b858b10-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal\" (UID: \"3f537d23616ee809add516752b858b10\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:44.393442 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:44.393368    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:45.548226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:45.548204    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:45.552074 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:45.552060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:45.570052 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:45.570034    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:45.670391 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:45.670356    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:45.770879 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:45.770807    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:45.871401 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:45.871376    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:45.916836 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:45.916808    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 17:24:45.917270 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:45.916966    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 17:24:45.971902 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:45.971872    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:46.005952 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.005913    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:46.044439 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.044363    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 17:19:45 +0000 UTC" deadline="2028-03-12 10:52:57.076233526 +0000 UTC"
Sep 16 17:24:46.044439 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.044400    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13025h28m11.031836154s"
Sep 16 17:24:46.072564 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:46.072528    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:46.088989 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.088961    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 17:24:46.090169 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.090153    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 17:24:46.090233 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.090184    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 17:24:46.090233 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.090195    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeHasSufficientPID"
Sep 16 17:24:46.146670 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:46.146639    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f537d23616ee809add516752b858b10.slice/crio-5eb873faf54d1c8fdc9c243eda17bc94b30fbcd27178305e85203b22ed4c824a WatchSource:0}: Error finding container 5eb873faf54d1c8fdc9c243eda17bc94b30fbcd27178305e85203b22ed4c824a: Status 404 returned error can't find the container with id 5eb873faf54d1c8fdc9c243eda17bc94b30fbcd27178305e85203b22ed4c824a
Sep 16 17:24:46.147566 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:46.147463    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d152f82d086cc4921e20e44fe174b4.slice/crio-ef6d450c0765c879cb9237f7cf8fa2a61347f942923e7a1fa1de0d4f690bcd98 WatchSource:0}: Error finding container ef6d450c0765c879cb9237f7cf8fa2a61347f942923e7a1fa1de0d4f690bcd98: Status 404 returned error can't find the container with id ef6d450c0765c879cb9237f7cf8fa2a61347f942923e7a1fa1de0d4f690bcd98
Sep 16 17:24:46.154985 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.154968    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:24:46.173131 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:46.173109    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:46.273651 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:46.273576    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-156.ec2.internal\" not found"
Sep 16 17:24:46.287145 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.287127    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:46.327525 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.327473    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:46.377320 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.377283    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:46.396469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.396446    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:46.397257 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.397244    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal"
Sep 16 17:24:46.420556 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.420529    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 17:24:46.958524 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.958492    2560 apiserver.go:52] "Watching apiserver"
Sep 16 17:24:46.964465 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.964442    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 17:24:46.965799 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.965764    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-w5vt2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal","openshift-multus/multus-5g7vl","openshift-multus/network-metrics-daemon-bfp6l","openshift-cluster-node-tuning-operator/tuned-48mxf","openshift-dns/node-resolver-q426q","openshift-multus/multus-additional-cni-plugins-jnlxg","openshift-network-diagnostics/network-check-target-mfjgf","openshift-network-operator/iptables-alerter-vhdds","openshift-ovn-kubernetes/ovnkube-node-w7tlg","kube-system/global-pull-secret-syncer-5sjzg","kube-system/konnectivity-agent-s2f66","kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"]
Sep 16 17:24:46.968288 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.968267    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.970529 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.970465    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:46.970647 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:46.970545    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:46.972705 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.972679    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.974941 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.974922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:46.975069 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:46.974991    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:24:46.976571 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.976548    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 17:24:46.977163 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.976550    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 17:24:46.977615 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.977579    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 17:24:46.977925 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.977906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 17:24:46.978094 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.978081    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 17:24:46.978223 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.978204    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-n7cgl\""
Sep 16 17:24:46.978305 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.978228    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8ksm4\""
Sep 16 17:24:46.978707 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.978429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.978707 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.978439    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 17:24:46.978707 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.978433    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 17:24:46.980432 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.980114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:46.982022 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.982002    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jhh58\""
Sep 16 17:24:46.982702 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.982454    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:46.984888 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.984872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:46.987134 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.987116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:46.987233 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:46.987175    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:46.989659 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.989640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-cnibin\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.989756 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.989679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0640143a-e0a6-4db6-85be-93f4cf337db9-kubelet-config\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:46.990203 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.990181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-netns\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.990270 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.990233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-cni-multus\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.990321 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.990269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:46.990321 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.990302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckz8k\" (UniqueName: \"kubernetes.io/projected/4304ee43-101e-44b9-bfc7-87b2e0a9273f-kube-api-access-ckz8k\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:46.990430 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.990357    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:46.991197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.990333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.991312 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991227    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-hostroot\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991373 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-lib-modules\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.991373 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-var-lib-kubelet\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.991467 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0640143a-e0a6-4db6-85be-93f4cf337db9-dbus\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:46.991467 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-k8s-cni-cncf-io\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991556 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991467    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-kubelet\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991556 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-run\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.991556 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-tuned\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.991769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-system-cni-dir\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.991769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w748q\" (UniqueName: \"kubernetes.io/projected/55dcaec8-4335-427e-a986-304d6cc2166c-kube-api-access-w748q\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991856 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-host\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.991856 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-system-cni-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991946 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-cni-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991946 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-cnibin\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.991946 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991934    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-cni-bin\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.992078 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-kubernetes\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992078 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.991989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysctl-conf\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992078 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-systemd\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992215 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9qjn\" (UniqueName: \"kubernetes.io/projected/ac928175-f4b0-4331-b4f2-7f4d51728dd8-kube-api-access-v9qjn\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992261 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992220    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.992261 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992251    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.992347 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbtp\" (UniqueName: \"kubernetes.io/projected/25a56d28-4fc5-49fb-9375-0a2ee9124104-kube-api-access-gsbtp\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.992347 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55dcaec8-4335-427e-a986-304d6cc2166c-cni-binary-copy\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.992347 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysconfig\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992477 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac928175-f4b0-4331-b4f2-7f4d51728dd8-tmp\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992477 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:46.992564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-conf-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.992564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysctl-d\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992530    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:46.992564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-socket-dir-parent\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.992564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55dcaec8-4335-427e-a986-304d6cc2166c-multus-daemon-config\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.992790 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-etc-kubernetes\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.992790 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992621    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-modprobe-d\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992790 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992653    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-sys\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:46.992790 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.992670    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-multus-certs\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:46.994954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.994642    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 17:24:46.994954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.994785    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:46.994954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.994848    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 17:24:46.994954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.994858    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:46.994954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.994904    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lnjxd\""
Sep 16 17:24:46.994954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.994848    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 17:24:46.995279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.995048    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 17:24:46.995279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.995055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:46.995279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.995215    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:24:46.995279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.995046    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 17:24:46.995453 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.995349    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wwr9h\""
Sep 16 17:24:46.995501 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.995474    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-749hx\""
Sep 16 17:24:46.997087 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.997008    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:46.997666 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.997645    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 17:24:46.997742 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.997683    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 17:24:46.997880 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.997864    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 17:24:46.997946 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:46.997899    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 17:24:47.007287 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.007264    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 17:24:47.007385 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.007315    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 17:24:47.011138 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.010977    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 17:24:47.011506 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.011470    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dz7gd\""
Sep 16 17:24:47.012547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.011689    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 17:24:47.012547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.011886    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvgbv\""
Sep 16 17:24:47.012547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.011951    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 17:24:47.012547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.012114    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mwpng\""
Sep 16 17:24:47.012547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.012253    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 17:24:47.012911 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.012669    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 17:24:47.013363 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.013202    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 17:24:47.013363 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.013253    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 17:24:47.034669 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.034649    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 17:24:47.078261 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.078236    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 17:24:47.093042 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-conf-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093167 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysctl-d\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093167 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c76ab09-5936-4425-94f8-dd03433ecfa9-serviceca\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.093167 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.093167 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093120    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-conf-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093167 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-socket-dir-parent\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55dcaec8-4335-427e-a986-304d6cc2166c-multus-daemon-config\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-sys\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-socket-dir-parent\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093230    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-kubelet-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysctl-d\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093267    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-sys\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093254    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-registration-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-sys-fs\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-cnibin\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093388    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-cnibin\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.093401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093392    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-netns\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-cni-multus\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckz8k\" (UniqueName: \"kubernetes.io/projected/4304ee43-101e-44b9-bfc7-87b2e0a9273f-kube-api-access-ckz8k\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093498    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-netns\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-tmp-dir\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093520    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-cni-multus\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk7zr\" (UniqueName: \"kubernetes.io/projected/857f9c7c-8310-432f-b90c-10aab8f2ea3c-kube-api-access-gk7zr\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-systemd\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-socket-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-var-lib-kubelet\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-run-netns\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0640143a-e0a6-4db6-85be-93f4cf337db9-dbus\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-kubelet\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-run\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-cni-binary-copy\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0640143a-e0a6-4db6-85be-93f4cf337db9-dbus\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-var-lib-kubelet\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.093935 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-kubelet\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-systemd-units\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-kubelet\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-run\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovnkube-config\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-etc-selinux\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w748q\" (UniqueName: \"kubernetes.io/projected/55dcaec8-4335-427e-a986-304d6cc2166c-kube-api-access-w748q\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.093998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-cni-bin\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-slash\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094067    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-cni-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysctl-conf\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094179    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-multus-cni-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysctl-conf\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094320    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-systemd\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.094597 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55dcaec8-4335-427e-a986-304d6cc2166c-multus-daemon-config\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-systemd\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094641    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9qjn\" (UniqueName: \"kubernetes.io/projected/ac928175-f4b0-4331-b4f2-7f4d51728dd8-kube-api-access-v9qjn\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr4l2\" (UniqueName: \"kubernetes.io/projected/9f4909e8-270b-4cf4-989b-b948c82cd4df-kube-api-access-fr4l2\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094705    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/857f9c7c-8310-432f-b90c-10aab8f2ea3c-host-slash\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55dcaec8-4335-427e-a986-304d6cc2166c-cni-binary-copy\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysconfig\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac928175-f4b0-4331-b4f2-7f4d51728dd8-tmp\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094836    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-var-lib-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/254141ee-0a25-484f-b28a-0b1485f4ffc9-agent-certs\") pod \"konnectivity-agent-s2f66\" (UID: \"254141ee-0a25-484f-b28a-0b1485f4ffc9\") " pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094895    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-device-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094920    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-sysconfig\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-log-socket\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-cni-netd\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.094999    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovnkube-script-lib\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-etc-kubernetes\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.095034    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-modprobe-d\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-etc-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-etc-kubernetes\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55dcaec8-4335-427e-a986-304d6cc2166c-cni-binary-copy\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.095163    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:47.595090866 +0000 UTC m=+3.004208112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-modprobe-d\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-multus-certs\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-multus-certs\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0640143a-e0a6-4db6-85be-93f4cf337db9-kubelet-config\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0640143a-e0a6-4db6-85be-93f4cf337db9-kubelet-config\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-ovn\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095349    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-node-log\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.095373    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:47.095992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095374    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/254141ee-0a25-484f-b28a-0b1485f4ffc9-konnectivity-ca\") pod \"konnectivity-agent-s2f66\" (UID: \"254141ee-0a25-484f-b28a-0b1485f4ffc9\") " pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.095428    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:47.595411836 +0000 UTC m=+3.004529082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095450    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095506    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-hostroot\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-hostroot\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-lib-modules\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c76ab09-5936-4425-94f8-dd03433ecfa9-host\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-k8s-cni-cncf-io\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-tuned\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovn-node-metrics-cert\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-run-k8s-cni-cncf-io\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdn4f\" (UniqueName: \"kubernetes.io/projected/8367012f-1e90-4eab-8b74-b4c90276be37-kube-api-access-qdn4f\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-lib-modules\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-system-cni-dir\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-host\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.096647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-system-cni-dir\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-host\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095907    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/857f9c7c-8310-432f-b90c-10aab8f2ea3c-iptables-alerter-script\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.095944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-system-cni-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-cnibin\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-system-cni-dir\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-cni-bin\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-kubernetes\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-cnibin\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-hosts-file\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55dcaec8-4335-427e-a986-304d6cc2166c-host-var-lib-cni-bin\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lkl\" (UniqueName: \"kubernetes.io/projected/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-kube-api-access-z4lkl\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-kubernetes\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5fc9\" (UniqueName: \"kubernetes.io/projected/7c76ab09-5936-4425-94f8-dd03433ecfa9-kube-api-access-x5fc9\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.097386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.098197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbtp\" (UniqueName: \"kubernetes.io/projected/25a56d28-4fc5-49fb-9375-0a2ee9124104-kube-api-access-gsbtp\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.098197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/25a56d28-4fc5-49fb-9375-0a2ee9124104-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.098197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-env-overrides\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.098197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.096749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/25a56d28-4fc5-49fb-9375-0a2ee9124104-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.099181 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.099158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac928175-f4b0-4331-b4f2-7f4d51728dd8-tmp\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.099285 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.099196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ac928175-f4b0-4331-b4f2-7f4d51728dd8-etc-tuned\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.110789 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.110394    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckz8k\" (UniqueName: \"kubernetes.io/projected/4304ee43-101e-44b9-bfc7-87b2e0a9273f-kube-api-access-ckz8k\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:47.110789 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.110398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbtp\" (UniqueName: \"kubernetes.io/projected/25a56d28-4fc5-49fb-9375-0a2ee9124104-kube-api-access-gsbtp\") pod \"multus-additional-cni-plugins-jnlxg\" (UID: \"25a56d28-4fc5-49fb-9375-0a2ee9124104\") " pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.114217 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.114193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w748q\" (UniqueName: \"kubernetes.io/projected/55dcaec8-4335-427e-a986-304d6cc2166c-kube-api-access-w748q\") pod \"multus-5g7vl\" (UID: \"55dcaec8-4335-427e-a986-304d6cc2166c\") " pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.115954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.115914    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal" event={"ID":"e1d152f82d086cc4921e20e44fe174b4","Type":"ContainerStarted","Data":"ef6d450c0765c879cb9237f7cf8fa2a61347f942923e7a1fa1de0d4f690bcd98"}
Sep 16 17:24:47.117202 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.117176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal" event={"ID":"3f537d23616ee809add516752b858b10","Type":"ContainerStarted","Data":"5eb873faf54d1c8fdc9c243eda17bc94b30fbcd27178305e85203b22ed4c824a"}
Sep 16 17:24:47.118609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.118574    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9qjn\" (UniqueName: \"kubernetes.io/projected/ac928175-f4b0-4331-b4f2-7f4d51728dd8-kube-api-access-v9qjn\") pod \"tuned-48mxf\" (UID: \"ac928175-f4b0-4331-b4f2-7f4d51728dd8\") " pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.196551 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196521    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-db5xq"
Sep 16 17:24:47.196728 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196728 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-cni-bin\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196728 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-slash\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr4l2\" (UniqueName: \"kubernetes.io/projected/9f4909e8-270b-4cf4-989b-b948c82cd4df-kube-api-access-fr4l2\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-cni-bin\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-slash\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196817    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-run-ovn-kubernetes\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/857f9c7c-8310-432f-b90c-10aab8f2ea3c-host-slash\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/857f9c7c-8310-432f-b90c-10aab8f2ea3c-host-slash\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-var-lib-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.196919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/254141ee-0a25-484f-b28a-0b1485f4ffc9-agent-certs\") pod \"konnectivity-agent-s2f66\" (UID: \"254141ee-0a25-484f-b28a-0b1485f4ffc9\") " pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-device-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-log-socket\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.196996    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-var-lib-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-log-socket\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-device-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-cni-netd\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovnkube-script-lib\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-etc-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-cni-netd\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-ovn\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-node-log\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197224    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/254141ee-0a25-484f-b28a-0b1485f4ffc9-konnectivity-ca\") pod \"konnectivity-agent-s2f66\" (UID: \"254141ee-0a25-484f-b28a-0b1485f4ffc9\") " pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c76ab09-5936-4425-94f8-dd03433ecfa9-host\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-ovn\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovn-node-metrics-cert\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdn4f\" (UniqueName: \"kubernetes.io/projected/8367012f-1e90-4eab-8b74-b4c90276be37-kube-api-access-qdn4f\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.197429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-etc-openvswitch\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/857f9c7c-8310-432f-b90c-10aab8f2ea3c-iptables-alerter-script\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197406    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-hosts-file\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197434    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lkl\" (UniqueName: \"kubernetes.io/projected/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-kube-api-access-z4lkl\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5fc9\" (UniqueName: \"kubernetes.io/projected/7c76ab09-5936-4425-94f8-dd03433ecfa9-kube-api-access-x5fc9\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-env-overrides\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c76ab09-5936-4425-94f8-dd03433ecfa9-serviceca\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-kubelet-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-registration-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-sys-fs\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-tmp-dir\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk7zr\" (UniqueName: \"kubernetes.io/projected/857f9c7c-8310-432f-b90c-10aab8f2ea3c-kube-api-access-gk7zr\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-systemd\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-socket-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-run-netns\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-kubelet\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.198245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovnkube-script-lib\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-systemd-units\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-systemd-units\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovnkube-config\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-etc-selinux\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/857f9c7c-8310-432f-b90c-10aab8f2ea3c-iptables-alerter-script\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-hosts-file\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/254141ee-0a25-484f-b28a-0b1485f4ffc9-konnectivity-ca\") pod \"konnectivity-agent-s2f66\" (UID: \"254141ee-0a25-484f-b28a-0b1485f4ffc9\") " pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-node-log\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-etc-selinux\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.197980    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-sys-fs\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198119    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c76ab09-5936-4425-94f8-dd03433ecfa9-host\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-socket-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-tmp-dir\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-run-netns\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-host-kubelet\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9f4909e8-270b-4cf4-989b-b948c82cd4df-run-systemd\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198407    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-kubelet-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198431    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovnkube-config\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8367012f-1e90-4eab-8b74-b4c90276be37-registration-dir\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9f4909e8-270b-4cf4-989b-b948c82cd4df-env-overrides\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.198712    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c76ab09-5936-4425-94f8-dd03433ecfa9-serviceca\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.199768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f4909e8-270b-4cf4-989b-b948c82cd4df-ovn-node-metrics-cert\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.199863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.199823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/254141ee-0a25-484f-b28a-0b1485f4ffc9-agent-certs\") pod \"konnectivity-agent-s2f66\" (UID: \"254141ee-0a25-484f-b28a-0b1485f4ffc9\") " pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.221340 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.221260    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-db5xq"
Sep 16 17:24:47.223084 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.223061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr4l2\" (UniqueName: \"kubernetes.io/projected/9f4909e8-270b-4cf4-989b-b948c82cd4df-kube-api-access-fr4l2\") pod \"ovnkube-node-w7tlg\" (UID: \"9f4909e8-270b-4cf4-989b-b948c82cd4df\") " pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.229214 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.229193    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk7zr\" (UniqueName: \"kubernetes.io/projected/857f9c7c-8310-432f-b90c-10aab8f2ea3c-kube-api-access-gk7zr\") pod \"iptables-alerter-vhdds\" (UID: \"857f9c7c-8310-432f-b90c-10aab8f2ea3c\") " pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.231986 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.231959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdn4f\" (UniqueName: \"kubernetes.io/projected/8367012f-1e90-4eab-8b74-b4c90276be37-kube-api-access-qdn4f\") pod \"aws-ebs-csi-driver-node-whggl\" (UID: \"8367012f-1e90-4eab-8b74-b4c90276be37\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.232138 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.232111    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5fc9\" (UniqueName: \"kubernetes.io/projected/7c76ab09-5936-4425-94f8-dd03433ecfa9-kube-api-access-x5fc9\") pod \"node-ca-w5vt2\" (UID: \"7c76ab09-5936-4425-94f8-dd03433ecfa9\") " pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.235432 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.235411    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:47.235522 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.235438    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:47.235522 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.235452    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:47.235522 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.235512    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:47.735499122 +0000 UTC m=+3.144616348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:47.239383 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.239362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lkl\" (UniqueName: \"kubernetes.io/projected/3a3f53ed-df0f-4f40-94c1-0fdacaa54f70-kube-api-access-z4lkl\") pod \"node-resolver-q426q\" (UID: \"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70\") " pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.240402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.240384    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 17:24:47.282172 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.282143    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jnlxg"
Sep 16 17:24:47.290329 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.290305    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5g7vl"
Sep 16 17:24:47.302292 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.302274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-48mxf"
Sep 16 17:24:47.307784 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.307759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q426q"
Sep 16 17:24:47.316183 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.316162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vhdds"
Sep 16 17:24:47.322313 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.322297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl"
Sep 16 17:24:47.328985 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.328971    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:24:47.334610 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.334578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w5vt2"
Sep 16 17:24:47.339248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.339219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:24:47.600309 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.600241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:47.600309 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.600275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:47.600515 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.600372    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:47.600515 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.600391    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:47.600515 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.600420    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:48.600406669 +0000 UTC m=+4.009523911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:47.600515 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.600447    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:48.600431713 +0000 UTC m=+4.009548942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:47.801817 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:47.801786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:47.801997 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.801978    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:47.802049 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.802004    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:47.802049 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.802020    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:47.802127 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:47.802077    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:48.802058596 +0000 UTC m=+4.211175825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:47.986098 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:47.985925    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3f53ed_df0f_4f40_94c1_0fdacaa54f70.slice/crio-3942b4359e2b3a87ec0c647504e6f672630697f3c2bee083d5b76a8f12d1ac23 WatchSource:0}: Error finding container 3942b4359e2b3a87ec0c647504e6f672630697f3c2bee083d5b76a8f12d1ac23: Status 404 returned error can't find the container with id 3942b4359e2b3a87ec0c647504e6f672630697f3c2bee083d5b76a8f12d1ac23
Sep 16 17:24:47.987760 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:47.987738    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dcaec8_4335_427e_a986_304d6cc2166c.slice/crio-39f6769c3ed5a59bf64f400701883b4881b2bbe6b8bb38d1d86e04e24ca08faf WatchSource:0}: Error finding container 39f6769c3ed5a59bf64f400701883b4881b2bbe6b8bb38d1d86e04e24ca08faf: Status 404 returned error can't find the container with id 39f6769c3ed5a59bf64f400701883b4881b2bbe6b8bb38d1d86e04e24ca08faf
Sep 16 17:24:47.988983 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:47.988916    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254141ee_0a25_484f_b28a_0b1485f4ffc9.slice/crio-1c8edcd6ce99bc1fdc990640e917d35a6a08c066f2433e1c3484942c3caad47b WatchSource:0}: Error finding container 1c8edcd6ce99bc1fdc990640e917d35a6a08c066f2433e1c3484942c3caad47b: Status 404 returned error can't find the container with id 1c8edcd6ce99bc1fdc990640e917d35a6a08c066f2433e1c3484942c3caad47b
Sep 16 17:24:47.991384 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:47.991351    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8367012f_1e90_4eab_8b74_b4c90276be37.slice/crio-2732d1366b6961a458fac5e7c33e9960028813dcc5126c1e1952dedbab60a3c3 WatchSource:0}: Error finding container 2732d1366b6961a458fac5e7c33e9960028813dcc5126c1e1952dedbab60a3c3: Status 404 returned error can't find the container with id 2732d1366b6961a458fac5e7c33e9960028813dcc5126c1e1952dedbab60a3c3
Sep 16 17:24:47.992045 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:47.992023    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c76ab09_5936_4425_94f8_dd03433ecfa9.slice/crio-83e8b09fae9de19368a29184715ee18d2912e6765bd40a205a316ff40ee276b9 WatchSource:0}: Error finding container 83e8b09fae9de19368a29184715ee18d2912e6765bd40a205a316ff40ee276b9: Status 404 returned error can't find the container with id 83e8b09fae9de19368a29184715ee18d2912e6765bd40a205a316ff40ee276b9
Sep 16 17:24:47.992894 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:47.992804    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4909e8_270b_4cf4_989b_b948c82cd4df.slice/crio-8e4bfecb2719ebd7f25a2d100d285c52395b3c98ae2458b530f88483cf2c46f7 WatchSource:0}: Error finding container 8e4bfecb2719ebd7f25a2d100d285c52395b3c98ae2458b530f88483cf2c46f7: Status 404 returned error can't find the container with id 8e4bfecb2719ebd7f25a2d100d285c52395b3c98ae2458b530f88483cf2c46f7
Sep 16 17:24:48.014723 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:48.014698    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857f9c7c_8310_432f_b90c_10aab8f2ea3c.slice/crio-c62b8a0523ac6ce62dcfada8449c815a9ce4fdcc88cdd91dcf03372cf3c3cddf WatchSource:0}: Error finding container c62b8a0523ac6ce62dcfada8449c815a9ce4fdcc88cdd91dcf03372cf3c3cddf: Status 404 returned error can't find the container with id c62b8a0523ac6ce62dcfada8449c815a9ce4fdcc88cdd91dcf03372cf3c3cddf
Sep 16 17:24:48.015872 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:24:48.015632    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac928175_f4b0_4331_b4f2_7f4d51728dd8.slice/crio-47937d4d77a9a95e1e601651a43384ce5039b1018ca1c9567f7d430f181cd7dd WatchSource:0}: Error finding container 47937d4d77a9a95e1e601651a43384ce5039b1018ca1c9567f7d430f181cd7dd: Status 404 returned error can't find the container with id 47937d4d77a9a95e1e601651a43384ce5039b1018ca1c9567f7d430f181cd7dd
Sep 16 17:24:48.111943 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.111922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:48.112050 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.112010    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:48.112050 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.112014    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:24:48.112050 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.112037    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:48.112195 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.112090    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:48.112195 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.112170    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:48.120200 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.120165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2f66" event={"ID":"254141ee-0a25-484f-b28a-0b1485f4ffc9","Type":"ContainerStarted","Data":"1c8edcd6ce99bc1fdc990640e917d35a6a08c066f2433e1c3484942c3caad47b"}
Sep 16 17:24:48.121471 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.121451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q426q" event={"ID":"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70","Type":"ContainerStarted","Data":"3942b4359e2b3a87ec0c647504e6f672630697f3c2bee083d5b76a8f12d1ac23"}
Sep 16 17:24:48.122666 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.122644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl" event={"ID":"8367012f-1e90-4eab-8b74-b4c90276be37","Type":"ContainerStarted","Data":"2732d1366b6961a458fac5e7c33e9960028813dcc5126c1e1952dedbab60a3c3"}
Sep 16 17:24:48.123856 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.123837    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5g7vl" event={"ID":"55dcaec8-4335-427e-a986-304d6cc2166c","Type":"ContainerStarted","Data":"39f6769c3ed5a59bf64f400701883b4881b2bbe6b8bb38d1d86e04e24ca08faf"}
Sep 16 17:24:48.125673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.125650    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal" event={"ID":"e1d152f82d086cc4921e20e44fe174b4","Type":"ContainerStarted","Data":"0a1a876931bca035ada8e395a33d18a16add2aef279651b617e7f4cc7523d601"}
Sep 16 17:24:48.126917 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.126899    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerStarted","Data":"fc95a1a206b5ebfde318d1c290ea6a6d4b41c365d196042e28c3ee6dc46ce4f1"}
Sep 16 17:24:48.128073 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.128033    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w5vt2" event={"ID":"7c76ab09-5936-4425-94f8-dd03433ecfa9","Type":"ContainerStarted","Data":"83e8b09fae9de19368a29184715ee18d2912e6765bd40a205a316ff40ee276b9"}
Sep 16 17:24:48.129555 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.129536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-48mxf" event={"ID":"ac928175-f4b0-4331-b4f2-7f4d51728dd8","Type":"ContainerStarted","Data":"47937d4d77a9a95e1e601651a43384ce5039b1018ca1c9567f7d430f181cd7dd"}
Sep 16 17:24:48.130941 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.130918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhdds" event={"ID":"857f9c7c-8310-432f-b90c-10aab8f2ea3c","Type":"ContainerStarted","Data":"c62b8a0523ac6ce62dcfada8449c815a9ce4fdcc88cdd91dcf03372cf3c3cddf"}
Sep 16 17:24:48.132124 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.132105    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"8e4bfecb2719ebd7f25a2d100d285c52395b3c98ae2458b530f88483cf2c46f7"}
Sep 16 17:24:48.151003 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.150952    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-156.ec2.internal" podStartSLOduration=2.150936653 podStartE2EDuration="2.150936653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:48.150899738 +0000 UTC m=+3.560016987" watchObservedRunningTime="2026-09-16 17:24:48.150936653 +0000 UTC m=+3.560053901"
Sep 16 17:24:48.226320 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.226244    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:47 +0000 UTC" deadline="2028-05-10 11:51:05.433462754 +0000 UTC"
Sep 16 17:24:48.226320 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.226270    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14442h26m17.207195452s"
Sep 16 17:24:48.606603 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.606509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:48.606764 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.606624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:48.606764 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.606748    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:48.606875 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.606811    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:50.606792356 +0000 UTC m=+6.015909587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:48.607211 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.607191    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:48.607297 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.607234    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:50.60722367 +0000 UTC m=+6.016340896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:48.809085 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:48.809044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:48.809259 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.809227    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:48.809259 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.809248    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:48.809377 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.809260    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:48.809377 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:48.809317    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:50.809299712 +0000 UTC m=+6.218416942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:49.144863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:49.144822    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f537d23616ee809add516752b858b10" containerID="ef927f585a0db0de4ba33527870cfac5b7cec3f3ad15d9e315c61f2741f3a667" exitCode=0
Sep 16 17:24:49.145669 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:49.145328    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal" event={"ID":"3f537d23616ee809add516752b858b10","Type":"ContainerDied","Data":"ef927f585a0db0de4ba33527870cfac5b7cec3f3ad15d9e315c61f2741f3a667"}
Sep 16 17:24:49.226808 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:49.226734    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 17:19:47 +0000 UTC" deadline="2028-06-23 13:32:58.146354682 +0000 UTC"
Sep 16 17:24:49.226808 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:49.226771    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15500h8m8.919588625s"
Sep 16 17:24:50.112546 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.112475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:50.112729 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.112613    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:50.113004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.112986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:50.113119 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.113095    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:24:50.113183 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.113161    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:50.113276 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.113232    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:50.161560 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.161528    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal" event={"ID":"3f537d23616ee809add516752b858b10","Type":"ContainerStarted","Data":"84602ea7c4efad88066913d6cd2c86b05b5619f7c5ce1ee4937eb0a9a48dd734"}
Sep 16 17:24:50.629606 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.629558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:50.629769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.629627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:50.629769 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.629728    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:50.629892 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.629768    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:50.629892 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.629801    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.629781669 +0000 UTC m=+10.038898897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:50.629892 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.629821    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.629810853 +0000 UTC m=+10.038928084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:50.831232 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:50.831195    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:50.831430 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.831410    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:50.831529 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.831434    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:50.831529 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.831447    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:50.831529 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:50.831503    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:24:54.831485638 +0000 UTC m=+10.240602880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:52.112052 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:52.112016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:52.112518 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:52.112164    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:52.112518 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:52.112302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:52.112518 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:52.112395    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:52.112518 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:52.112420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:52.112518 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:52.112494    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:24:54.112458 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:54.112383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:54.112458 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:54.112424    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:54.112966 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:54.112383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:54.112966 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.112519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:54.112966 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.112670    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:24:54.112966 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.112754    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:54.664536 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:54.664493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:54.664731 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:54.664553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:54.664731 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.664700    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:54.664836 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.664760    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:02.664741414 +0000 UTC m=+18.073858640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:24:54.665165 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.665145    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:54.665311 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.665197    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:02.665182313 +0000 UTC m=+18.074299543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:24:54.865648 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:54.865610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:54.865851 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.865802    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:24:54.865851 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.865821    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:24:54.865851 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.865833    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:54.866034 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:54.865890    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:02.865871932 +0000 UTC m=+18.274989175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:24:56.111994 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:56.111929    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:56.111994 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:56.111956    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:56.112474 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:56.112066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:56.112474 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:56.112073    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:24:56.112474 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:56.112202    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:56.112474 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:56.112286    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:58.112330 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:58.112298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:24:58.112330 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:58.112316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:24:58.112330 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:24:58.112298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:24:58.112775 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:58.112407    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:24:58.112775 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:58.112460    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:24:58.112775 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:24:58.112526    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:00.112387 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:00.112355    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:00.112812 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:00.112503    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:00.112812 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:00.112510    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:00.112812 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:00.112582    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:00.112812 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:00.112511    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:00.112812 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:00.112657    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:02.112534 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:02.112500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:02.113073 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:02.112537    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:02.113073 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:02.112633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:02.113073 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.112644    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:02.113073 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.112710    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:02.113073 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.112789    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:02.722231 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:02.722193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:02.722231 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:02.722238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:02.722479 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.722345    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:02.722479 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.722352    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:02.722479 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.722406    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:18.722391223 +0000 UTC m=+34.131508452 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:02.722479 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.722420    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:18.722414152 +0000 UTC m=+34.131531377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:02.924431 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:02.924395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:02.924634 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.924566    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:02.924634 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.924585    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:02.924634 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.924608    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:02.924797 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:02.924661    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:18.924645569 +0000 UTC m=+34.333762814 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:04.112579 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:04.112544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:04.112981 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:04.112667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:04.112981 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:04.112786    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:04.112981 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:04.112669    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:04.112981 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:04.112672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:04.112981 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:04.112908    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:06.112272 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.112017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:06.112793 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.112084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:06.112793 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:06.112382    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:06.112793 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.112084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:06.112793 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:06.112447    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:06.112793 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:06.112557    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:06.192416 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.192393    2560 generic.go:356] "Generic (PLEG): container finished" podID="25a56d28-4fc5-49fb-9375-0a2ee9124104" containerID="b34d752794974452b98cdd16897103f3ed398e68abfeb8ec122c3a56ca9fc3f9" exitCode=0
Sep 16 17:25:06.192519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.192474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerDied","Data":"b34d752794974452b98cdd16897103f3ed398e68abfeb8ec122c3a56ca9fc3f9"}
Sep 16 17:25:06.193906 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.193885    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w5vt2" event={"ID":"7c76ab09-5936-4425-94f8-dd03433ecfa9","Type":"ContainerStarted","Data":"fe65963a68a4f72fcbc1c13727c2fae6b51a06cb7322d78f2c51bdcc5ed25b32"}
Sep 16 17:25:06.195552 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.195525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-48mxf" event={"ID":"ac928175-f4b0-4331-b4f2-7f4d51728dd8","Type":"ContainerStarted","Data":"03b4f2adc64b610db6c514ebe9fe49be41473cfbe0475ed6689f6a27c7cb53c9"}
Sep 16 17:25:06.198025 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.198007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"52b81a877ce608817f86fc7d0fcd9ca0513318d732209e4f28b7c2428b0bf089"}
Sep 16 17:25:06.198110 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.198090    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"0dd6a586a0b86469ac941b7f0620f547a476a049ede16590e5b4e06899274207"}
Sep 16 17:25:06.198180 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.198140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"d9fa844e3b0c41277b0a8fc8be640f659cf5793c2b947d6c028f929e142793d2"}
Sep 16 17:25:06.198232 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.198178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"b2c61635323a91cce1feb56a34a8c2c584c3653be1cea0b2e97766449065b058"}
Sep 16 17:25:06.199413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.199392    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2f66" event={"ID":"254141ee-0a25-484f-b28a-0b1485f4ffc9","Type":"ContainerStarted","Data":"e7dee5c96059e839dc7614739645d16e1c10f78dd61bc9d6eb2da90fb2ac32ba"}
Sep 16 17:25:06.200644 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.200628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q426q" event={"ID":"3a3f53ed-df0f-4f40-94c1-0fdacaa54f70","Type":"ContainerStarted","Data":"7100eec7eab59dc21a4d71774c6460d8db5f5c2966f45a6f2a79d934856aa057"}
Sep 16 17:25:06.201992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.201971    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl" event={"ID":"8367012f-1e90-4eab-8b74-b4c90276be37","Type":"ContainerStarted","Data":"b4ee959a5a01de6d9b8967a83e7db6429837ff93453fe6e1858e8e6da69f8dd2"}
Sep 16 17:25:06.203294 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.203269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5g7vl" event={"ID":"55dcaec8-4335-427e-a986-304d6cc2166c","Type":"ContainerStarted","Data":"b0b0bfc59c4d4808ede38ae71170c77d63f94f10f891d8e86e0d9876039c546c"}
Sep 16 17:25:06.223187 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.223152    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-156.ec2.internal" podStartSLOduration=20.223140264 podStartE2EDuration="20.223140264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:24:50.18807041 +0000 UTC m=+5.597187660" watchObservedRunningTime="2026-09-16 17:25:06.223140264 +0000 UTC m=+21.632257511"
Sep 16 17:25:06.299994 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.299952    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w5vt2" podStartSLOduration=7.679334579 podStartE2EDuration="21.299936285s" totalImagesPullingTime="13.620601706s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:06.299923865 +0000 UTC m=+21.709041113" watchObservedRunningTime="2026-09-16 17:25:06.299936285 +0000 UTC m=+21.709053533"
Sep 16 17:25:06.300262 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.300237    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q426q" podStartSLOduration=3.686463019 podStartE2EDuration="21.300229615s" totalImagesPullingTime="17.613766596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:06.263963519 +0000 UTC m=+21.673080768" watchObservedRunningTime="2026-09-16 17:25:06.300229615 +0000 UTC m=+21.709346924"
Sep 16 17:25:06.326149 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.326114    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s2f66" podStartSLOduration=7.682421788 podStartE2EDuration="21.326102135s" totalImagesPullingTime="13.643680347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:06.32448148 +0000 UTC m=+21.733598728" watchObservedRunningTime="2026-09-16 17:25:06.326102135 +0000 UTC m=+21.735219382"
Sep 16 17:25:06.381258 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.381155    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5g7vl" podStartSLOduration=3.594225689 podStartE2EDuration="21.381133599s" totalImagesPullingTime="17.78690791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:06.380514933 +0000 UTC m=+21.789632181" watchObservedRunningTime="2026-09-16 17:25:06.381133599 +0000 UTC m=+21.790250848"
Sep 16 17:25:06.745947 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.745919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:25:06.746555 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.746535    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:25:06.791192 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:06.791144    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-48mxf" podStartSLOduration=4.086542316 podStartE2EDuration="21.791127913s" totalImagesPullingTime="17.704585597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:06.450196331 +0000 UTC m=+21.859313581" watchObservedRunningTime="2026-09-16 17:25:06.791127913 +0000 UTC m=+22.200245162"
Sep 16 17:25:07.130496 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:07.130332    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 17:25:07.207727 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:07.207690    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vhdds" event={"ID":"857f9c7c-8310-432f-b90c-10aab8f2ea3c","Type":"ContainerStarted","Data":"057d4a26967cee595b596b48be64735a35bb09d1eb68e3a38b5838c2d0bdaae6"}
Sep 16 17:25:07.211443 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:07.211408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"38ff37b0bfd67acb06dbca31de2e884ccea1b375dfe1ceb37562218bf9683dd9"}
Sep 16 17:25:07.211610 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:07.211475    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"d1c282b2f311d6d7fab39af9dbf8583ce570643ebc7ed403b2b66e86794d4aba"}
Sep 16 17:25:07.213690 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:07.213640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl" event={"ID":"8367012f-1e90-4eab-8b74-b4c90276be37","Type":"ContainerStarted","Data":"b9c492974508686c6e44a2d5936d7a5fb3eef594cf65a75fd7f98219f8d859ec"}
Sep 16 17:25:08.050423 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.050310    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T17:25:07.130495949Z","UUID":"cc6498c2-710b-4e3d-9615-afbfbc8c6d36","Handler":null,"Name":"","Endpoint":""}
Sep 16 17:25:08.051805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.051778    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 17:25:08.051805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.051809    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 17:25:08.112881 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.112853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:08.113029 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:08.113002    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:08.113120 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.113024    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:08.113120 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.113039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:08.113232 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:08.113127    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:08.113232 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:08.113201    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:08.218916 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.218887    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl" event={"ID":"8367012f-1e90-4eab-8b74-b4c90276be37","Type":"ContainerStarted","Data":"89d08e4ed3f2ed0d26a87c7831d05566152c5d2f209aa0de3621c3296c76cd16"}
Sep 16 17:25:08.218916 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.218917    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 17:25:08.252838 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:08.252792    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vhdds" podStartSLOduration=5.54702349 podStartE2EDuration="23.252768755s" totalImagesPullingTime="17.705745265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:07.233205681 +0000 UTC m=+22.642322935" watchObservedRunningTime="2026-09-16 17:25:08.252768755 +0000 UTC m=+23.661886022"
Sep 16 17:25:09.226154 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:09.226122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"971523e60821960b7d70172e2198a050d8b0f408acf6ed2f53f87921653010d4"}
Sep 16 17:25:09.322009 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:09.321970    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:25:09.322192 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:09.322099    2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 17:25:09.322634 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:09.322608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s2f66"
Sep 16 17:25:09.351878 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:09.351825    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-whggl" podStartSLOduration=4.774988496 podStartE2EDuration="24.351809169s" totalImagesPullingTime="19.576820673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:08.253745277 +0000 UTC m=+23.662862529" watchObservedRunningTime="2026-09-16 17:25:09.351809169 +0000 UTC m=+24.760926416"
Sep 16 17:25:10.112512 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:10.112448    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:10.112690 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:10.112601    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:10.112690 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:10.112603    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:10.112795 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:10.112606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:10.112795 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:10.112715    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:10.112873 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:10.112800    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:11.232322 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.232158    2560 generic.go:356] "Generic (PLEG): container finished" podID="25a56d28-4fc5-49fb-9375-0a2ee9124104" containerID="13a1c35ab2e9d1d608b323bc9f6b09b181f077c8e6181313a328bc5b0784b097" exitCode=0
Sep 16 17:25:11.232809 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.232222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerDied","Data":"13a1c35ab2e9d1d608b323bc9f6b09b181f077c8e6181313a328bc5b0784b097"}
Sep 16 17:25:11.236307 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.236287    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" event={"ID":"9f4909e8-270b-4cf4-989b-b948c82cd4df","Type":"ContainerStarted","Data":"87366e643fb1f30f4577528123ccbb58c077d83670d7cf86dbc9383d77fad16e"}
Sep 16 17:25:11.236571 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.236553    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:25:11.236679 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.236622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:25:11.251488 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.251467    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:25:11.321457 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:11.321407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg" podStartSLOduration=8.609258888 podStartE2EDuration="26.32139148s" totalImagesPullingTime="17.712132592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:11.31949037 +0000 UTC m=+26.728607618" watchObservedRunningTime="2026-09-16 17:25:11.32139148 +0000 UTC m=+26.730508728"
Sep 16 17:25:12.112122 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.112046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:12.112122 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.112075    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:12.112328 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:12.112153    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:12.112328 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.112199    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:12.112328 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:12.112276    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:12.112461 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:12.112356    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:12.240039 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.240008    2560 generic.go:356] "Generic (PLEG): container finished" podID="25a56d28-4fc5-49fb-9375-0a2ee9124104" containerID="1e868f12916008ad84cb0a5b9edac07bb1e307d2bcea0a963e42724d7e8e9096" exitCode=0
Sep 16 17:25:12.240773 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.240082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerDied","Data":"1e868f12916008ad84cb0a5b9edac07bb1e307d2bcea0a963e42724d7e8e9096"}
Sep 16 17:25:12.240773 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.240514    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:25:12.254230 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.254210    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:25:12.743057 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.742974    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5sjzg"]
Sep 16 17:25:12.743204 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.743142    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:12.743266 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:12.743249    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:12.743686 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.743658    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfjgf"]
Sep 16 17:25:12.743769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.743746    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:12.743832 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:12.743815    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:12.767987 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.767964    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bfp6l"]
Sep 16 17:25:12.768122 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:12.768072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:12.768192 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:12.768175    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:13.244737 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:13.244541    2560 generic.go:356] "Generic (PLEG): container finished" podID="25a56d28-4fc5-49fb-9375-0a2ee9124104" containerID="fe55e6f2b653d2840802e8de6e39b45018bb55f2885f3d8a3388528fad37afda" exitCode=0
Sep 16 17:25:13.245427 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:13.244626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerDied","Data":"fe55e6f2b653d2840802e8de6e39b45018bb55f2885f3d8a3388528fad37afda"}
Sep 16 17:25:14.112684 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:14.112651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:14.112887 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:14.112651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:14.112887 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:14.112787    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:14.112887 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:14.112823    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:15.115936 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:15.115901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:15.116343 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:15.116031    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:16.112755 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:16.112721    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:16.112979 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:16.112838    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:16.112979 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:16.112900    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:16.113093 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:16.113019    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:17.112431 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:17.112398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:17.112899 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:17.112525    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfjgf" podUID="a4de0343-408d-44a6-8319-b4d446c75750"
Sep 16 17:25:18.111819 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.111791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:18.112111 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.111791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:18.112111 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.111920    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5sjzg" podUID="0640143a-e0a6-4db6-85be-93f4cf337db9"
Sep 16 17:25:18.112111 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.111963    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:25:18.262739 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.262709    2560 generic.go:356] "Generic (PLEG): container finished" podID="25a56d28-4fc5-49fb-9375-0a2ee9124104" containerID="2a890cce8f731e8c261837de514175dad5d62f0025c14d19cc85d410c0ea5f9e" exitCode=0
Sep 16 17:25:18.263043 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.262767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerDied","Data":"2a890cce8f731e8c261837de514175dad5d62f0025c14d19cc85d410c0ea5f9e"}
Sep 16 17:25:18.307676 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.307656    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-156.ec2.internal" event="NodeReady"
Sep 16 17:25:18.307829 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.307807    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 17:25:18.433918 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.433747    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qlnrn"]
Sep 16 17:25:18.438234 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.438219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.442499 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.442472    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 17:25:18.442499 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.442484    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 17:25:18.442709 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.442626    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 17:25:18.442767 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.442713    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dv5nc\""
Sep 16 17:25:18.450793 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.450773    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xpk25"]
Sep 16 17:25:18.456379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.456364    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.462389 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.462369    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 17:25:18.462478 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.462378    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 17:25:18.462478 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.462404    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qld9l\""
Sep 16 17:25:18.490579 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.490559    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xpk25"]
Sep 16 17:25:18.539973 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.539947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.539973 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.539973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41b04f03-bcb0-49ad-ac5d-a4cf82768494-cert\") pod \"ingress-canary-qlnrn\" (UID: \"41b04f03-bcb0-49ad-ac5d-a4cf82768494\") " pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.540139 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.540015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntdc\" (UniqueName: \"kubernetes.io/projected/18dd06e3-0da0-46da-b08e-d45b2396b86b-kube-api-access-4ntdc\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.540139 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.540063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdzx\" (UniqueName: \"kubernetes.io/projected/41b04f03-bcb0-49ad-ac5d-a4cf82768494-kube-api-access-gzdzx\") pod \"ingress-canary-qlnrn\" (UID: \"41b04f03-bcb0-49ad-ac5d-a4cf82768494\") " pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.540139 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.540117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd06e3-0da0-46da-b08e-d45b2396b86b-config-volume\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.540139 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.540133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18dd06e3-0da0-46da-b08e-d45b2396b86b-tmp-dir\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.579308 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.579281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qlnrn"]
Sep 16 17:25:18.640927 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.640878    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntdc\" (UniqueName: \"kubernetes.io/projected/18dd06e3-0da0-46da-b08e-d45b2396b86b-kube-api-access-4ntdc\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.640927 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.640912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdzx\" (UniqueName: \"kubernetes.io/projected/41b04f03-bcb0-49ad-ac5d-a4cf82768494-kube-api-access-gzdzx\") pod \"ingress-canary-qlnrn\" (UID: \"41b04f03-bcb0-49ad-ac5d-a4cf82768494\") " pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.641040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.640949    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd06e3-0da0-46da-b08e-d45b2396b86b-config-volume\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.641040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.640964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18dd06e3-0da0-46da-b08e-d45b2396b86b-tmp-dir\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.641040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.640996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.641040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.641017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41b04f03-bcb0-49ad-ac5d-a4cf82768494-cert\") pod \"ingress-canary-qlnrn\" (UID: \"41b04f03-bcb0-49ad-ac5d-a4cf82768494\") " pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.641224 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.641173    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:18.641277 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.641245    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:25:19.141222738 +0000 UTC m=+34.550339968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:18.641342 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.641296    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18dd06e3-0da0-46da-b08e-d45b2396b86b-tmp-dir\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.641502 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.641483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/18dd06e3-0da0-46da-b08e-d45b2396b86b-config-volume\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.645056 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.645031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/41b04f03-bcb0-49ad-ac5d-a4cf82768494-cert\") pod \"ingress-canary-qlnrn\" (UID: \"41b04f03-bcb0-49ad-ac5d-a4cf82768494\") " pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.653813 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.653790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdzx\" (UniqueName: \"kubernetes.io/projected/41b04f03-bcb0-49ad-ac5d-a4cf82768494-kube-api-access-gzdzx\") pod \"ingress-canary-qlnrn\" (UID: \"41b04f03-bcb0-49ad-ac5d-a4cf82768494\") " pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.670206 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.670184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntdc\" (UniqueName: \"kubernetes.io/projected/18dd06e3-0da0-46da-b08e-d45b2396b86b-kube-api-access-4ntdc\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:18.741987 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.741963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:18.742074 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.741994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:18.742112 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.742081    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:18.742112 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.742083    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:18.742172 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.742131    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret podName:0640143a-e0a6-4db6-85be-93f4cf337db9 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:50.742118466 +0000 UTC m=+66.151235696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret") pod "global-pull-secret-syncer-5sjzg" (UID: "0640143a-e0a6-4db6-85be-93f4cf337db9") : object "kube-system"/"original-pull-secret" not registered
Sep 16 17:25:18.742172 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.742151    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:25:50.742141567 +0000 UTC m=+66.151258792 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 17:25:18.797362 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.797346    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qlnrn"
Sep 16 17:25:18.943835 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:18.943762    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:18.943957 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.943888    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 17:25:18.943957 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.943910    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 17:25:18.943957 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.943920    2560 projected.go:196] Error preparing data for projected volume kube-api-access-87jq9 for pod openshift-network-diagnostics/network-check-target-mfjgf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:18.944057 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:18.943970    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9 podName:a4de0343-408d-44a6-8319-b4d446c75750 nodeName:}" failed. No retries permitted until 2026-09-16 17:25:50.943956174 +0000 UTC m=+66.353073400 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-87jq9" (UniqueName: "kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9") pod "network-check-target-mfjgf" (UID: "a4de0343-408d-44a6-8319-b4d446c75750") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 17:25:19.115238 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.115205    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:19.121316 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.121290    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:19.121316 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.121298    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhcq5\""
Sep 16 17:25:19.121566 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.121342    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:19.129332 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.129310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qlnrn"]
Sep 16 17:25:19.133955 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:25:19.133935    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b04f03_bcb0_49ad_ac5d_a4cf82768494.slice/crio-38baa4e1e43aaf940596c7437d57b405046322f8f4fc9b2430238f89a152f9cc WatchSource:0}: Error finding container 38baa4e1e43aaf940596c7437d57b405046322f8f4fc9b2430238f89a152f9cc: Status 404 returned error can't find the container with id 38baa4e1e43aaf940596c7437d57b405046322f8f4fc9b2430238f89a152f9cc
Sep 16 17:25:19.145948 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.145921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:19.146083 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:19.146066    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:19.146148 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:19.146132    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:25:20.14611178 +0000 UTC m=+35.555229007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:19.266435 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.266410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qlnrn" event={"ID":"41b04f03-bcb0-49ad-ac5d-a4cf82768494","Type":"ContainerStarted","Data":"38baa4e1e43aaf940596c7437d57b405046322f8f4fc9b2430238f89a152f9cc"}
Sep 16 17:25:19.269495 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.269469    2560 generic.go:356] "Generic (PLEG): container finished" podID="25a56d28-4fc5-49fb-9375-0a2ee9124104" containerID="646e9380053e8273a5a6a50212b1e94847d929b26e0a71f0a9878d6e31c9f2ec" exitCode=0
Sep 16 17:25:19.269648 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:19.269504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerDied","Data":"646e9380053e8273a5a6a50212b1e94847d929b26e0a71f0a9878d6e31c9f2ec"}
Sep 16 17:25:20.112017 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.111982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:20.112201 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.112167    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:20.116053 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.116031    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:20.116799 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.116781    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:20.117233 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.117208    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4r4\""
Sep 16 17:25:20.153681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.153654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:20.153817 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:20.153793    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:20.153863 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:20.153855    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:25:22.153837956 +0000 UTC m=+37.562955202 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:20.276035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.275819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" event={"ID":"25a56d28-4fc5-49fb-9375-0a2ee9124104","Type":"ContainerStarted","Data":"ab2a35a9a41205562da392d990cc85fb8e41a3de0733f7740724202b14fd62e1"}
Sep 16 17:25:20.316282 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:20.316227    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jnlxg" podStartSLOduration=7.920566373 podStartE2EDuration="35.316205636s" totalImagesPullingTime="27.395639263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:20.315980342 +0000 UTC m=+35.725097590" watchObservedRunningTime="2026-09-16 17:25:20.316205636 +0000 UTC m=+35.725322884"
Sep 16 17:25:22.168111 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:22.168075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:22.168524 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:22.168226    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:22.168524 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:22.168290    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:25:26.168276279 +0000 UTC m=+41.577393506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:22.282274 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:22.282248    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qlnrn" event={"ID":"41b04f03-bcb0-49ad-ac5d-a4cf82768494","Type":"ContainerStarted","Data":"7dbb430938916e903c93d104a856a8499146fc8b988a21f4e166df1ea9e99f64"}
Sep 16 17:25:22.325078 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:22.325033    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qlnrn" podStartSLOduration=1.866228491 podStartE2EDuration="4.325019526s" totalImagesPullingTime="2.458791035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:22.324769032 +0000 UTC m=+37.733886284" watchObservedRunningTime="2026-09-16 17:25:22.325019526 +0000 UTC m=+37.734136774"
Sep 16 17:25:26.195164 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:26.195123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:26.195565 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:26.195280    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:26.195565 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:26.195350    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:25:34.195331829 +0000 UTC m=+49.604449059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:34.249265 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:34.249226    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:34.249729 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:34.249385    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:34.249729 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:34.249453    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:25:50.249436499 +0000 UTC m=+65.658553724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:38.711578 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.711547    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"]
Sep 16 17:25:38.713290 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.713275    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.715472 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.715451    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 17:25:38.715998 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.715972    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"]
Sep 16 17:25:38.716097 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.716041    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 17:25:38.716097 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.716048    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 17:25:38.716555 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.716538    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 17:25:38.717796 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.717781    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.719770 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.719752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 17:25:38.719855 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.719804    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 17:25:38.720165 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.720148    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 17:25:38.720661 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.720399    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 17:25:38.728007 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.727990    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"]
Sep 16 17:25:38.731716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.731694    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"]
Sep 16 17:25:38.878978 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.878944    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.878978 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.878982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/049b2108-6157-40d6-ad2e-5ff5c1a1221d-tmp\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.879141 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879007    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwg27\" (UniqueName: \"kubernetes.io/projected/049b2108-6157-40d6-ad2e-5ff5c1a1221d-kube-api-access-gwg27\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.879141 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.879141 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879100    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-ca\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.879141 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-hub\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.879262 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r87p2\" (UniqueName: \"kubernetes.io/projected/79f9fecd-7d6a-4411-a444-92d747320af6-kube-api-access-r87p2\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.879262 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/049b2108-6157-40d6-ad2e-5ff5c1a1221d-klusterlet-config\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.879262 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.879190    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9fecd-7d6a-4411-a444-92d747320af6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/049b2108-6157-40d6-ad2e-5ff5c1a1221d-tmp\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.980327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwg27\" (UniqueName: \"kubernetes.io/projected/049b2108-6157-40d6-ad2e-5ff5c1a1221d-kube-api-access-gwg27\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.980469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-ca\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-hub\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r87p2\" (UniqueName: \"kubernetes.io/projected/79f9fecd-7d6a-4411-a444-92d747320af6-kube-api-access-r87p2\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/049b2108-6157-40d6-ad2e-5ff5c1a1221d-klusterlet-config\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.980469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9fecd-7d6a-4411-a444-92d747320af6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.980831 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.980806    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/049b2108-6157-40d6-ad2e-5ff5c1a1221d-tmp\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.981197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.981167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/79f9fecd-7d6a-4411-a444-92d747320af6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.984125 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.984096    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.984125 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.984122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.984279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.984096    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-hub\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.984279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.984098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/79f9fecd-7d6a-4411-a444-92d747320af6-ca\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.984279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.984180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/049b2108-6157-40d6-ad2e-5ff5c1a1221d-klusterlet-config\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:38.988752 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.988732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r87p2\" (UniqueName: \"kubernetes.io/projected/79f9fecd-7d6a-4411-a444-92d747320af6-kube-api-access-r87p2\") pod \"cluster-proxy-proxy-agent-57f46664dd-ml5n9\" (UID: \"79f9fecd-7d6a-4411-a444-92d747320af6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:38.991179 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:38.991156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwg27\" (UniqueName: \"kubernetes.io/projected/049b2108-6157-40d6-ad2e-5ff5c1a1221d-kube-api-access-gwg27\") pod \"klusterlet-addon-workmgr-8b8c68958-mq8qp\" (UID: \"049b2108-6157-40d6-ad2e-5ff5c1a1221d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:39.023676 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:39.023651    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:39.041048 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:39.041028    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:25:39.297803 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:39.297475    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"]
Sep 16 17:25:39.301756 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:25:39.301729    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049b2108_6157_40d6_ad2e_5ff5c1a1221d.slice/crio-53001e8b3a22391a5d467a4ea4ee407c3213853913ca3ee0a238e98985798b6a WatchSource:0}: Error finding container 53001e8b3a22391a5d467a4ea4ee407c3213853913ca3ee0a238e98985798b6a: Status 404 returned error can't find the container with id 53001e8b3a22391a5d467a4ea4ee407c3213853913ca3ee0a238e98985798b6a
Sep 16 17:25:39.316204 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:39.316158    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"]
Sep 16 17:25:39.318977 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:25:39.318952    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79f9fecd_7d6a_4411_a444_92d747320af6.slice/crio-a4ae962756b0c281f7ed837b94581af1117a77b75ea7bd17353b9549f0c316d1 WatchSource:0}: Error finding container a4ae962756b0c281f7ed837b94581af1117a77b75ea7bd17353b9549f0c316d1: Status 404 returned error can't find the container with id a4ae962756b0c281f7ed837b94581af1117a77b75ea7bd17353b9549f0c316d1
Sep 16 17:25:39.321640 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:39.321615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp" event={"ID":"049b2108-6157-40d6-ad2e-5ff5c1a1221d","Type":"ContainerStarted","Data":"53001e8b3a22391a5d467a4ea4ee407c3213853913ca3ee0a238e98985798b6a"}
Sep 16 17:25:40.326719 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:40.326673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" event={"ID":"79f9fecd-7d6a-4411-a444-92d747320af6","Type":"ContainerStarted","Data":"a4ae962756b0c281f7ed837b94581af1117a77b75ea7bd17353b9549f0c316d1"}
Sep 16 17:25:44.258820 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:44.258789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w7tlg"
Sep 16 17:25:45.340222 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.340181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp" event={"ID":"049b2108-6157-40d6-ad2e-5ff5c1a1221d","Type":"ContainerStarted","Data":"821eabc3764ff8c283aac40340b31b7bee20fff69fb11cb104c49be64c47029b"}
Sep 16 17:25:45.340707 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.340336    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:45.342206 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.342183    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp"
Sep 16 17:25:45.342468 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.342447    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" event={"ID":"79f9fecd-7d6a-4411-a444-92d747320af6","Type":"ContainerStarted","Data":"948e62e657f629d2c24de6de03d58b1f9e09e6460c9b3e5d84b75cc24890d59f"}
Sep 16 17:25:45.342523 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.342490    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" event={"ID":"79f9fecd-7d6a-4411-a444-92d747320af6","Type":"ContainerStarted","Data":"c74e00b84bdc4c892551185aa8bd7e8e9ffbfd8f1416573f457bbe729ccdf321"}
Sep 16 17:25:45.342523 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.342502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" event={"ID":"79f9fecd-7d6a-4411-a444-92d747320af6","Type":"ContainerStarted","Data":"f35eb55bf4834e3fcf3a09f7e29acae820e3e67f3a16fa2e0976a497822a8944"}
Sep 16 17:25:45.357774 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.357732    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8b8c68958-mq8qp" podStartSLOduration=1.9727317420000001 podStartE2EDuration="7.357718777s" totalImagesPullingTime="5.384987035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:45.35720748 +0000 UTC m=+60.766324731" watchObservedRunningTime="2026-09-16 17:25:45.357718777 +0000 UTC m=+60.766836041"
Sep 16 17:25:45.377673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:45.377635    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" podStartSLOduration=2.009235671 podStartE2EDuration="7.377623088s" totalImagesPullingTime="5.368387417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:45.377623508 +0000 UTC m=+60.786740759" watchObservedRunningTime="2026-09-16 17:25:45.377623088 +0000 UTC m=+60.786740336"
Sep 16 17:25:50.261043 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.260999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:25:50.261507 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:50.261140    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:25:50.261507 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:50.261200    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:26:22.261184871 +0000 UTC m=+97.670302097 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:25:50.764655 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.764579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:50.764850 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.764668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:25:50.766820 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.766800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 17:25:50.766988 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.766968    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 17:25:50.774911 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:50.774888    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:25:50.775004 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:25:50.774948    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:26:54.774933578 +0000 UTC m=+130.184050804 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : secret "metrics-daemon-secret" not found
Sep 16 17:25:50.776984 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.776966    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0640143a-e0a6-4db6-85be-93f4cf337db9-original-pull-secret\") pod \"global-pull-secret-syncer-5sjzg\" (UID: \"0640143a-e0a6-4db6-85be-93f4cf337db9\") " pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:50.966242 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.966192    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:50.968473 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.968451    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 17:25:50.979396 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.979379    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 17:25:50.990530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:50.990505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87jq9\" (UniqueName: \"kubernetes.io/projected/a4de0343-408d-44a6-8319-b4d446c75750-kube-api-access-87jq9\") pod \"network-check-target-mfjgf\" (UID: \"a4de0343-408d-44a6-8319-b4d446c75750\") " pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:51.024403 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:51.024353    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5sjzg"
Sep 16 17:25:51.226443 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:51.226414    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhcq5\""
Sep 16 17:25:51.235021 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:51.234997    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:51.290729 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:51.290655    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5sjzg"]
Sep 16 17:25:51.358244 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:51.358220    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5sjzg" event={"ID":"0640143a-e0a6-4db6-85be-93f4cf337db9","Type":"ContainerStarted","Data":"ef4d6713421ce1fbe0fd2bc6b501294d92c4393f6aa55e7328572c029ec7c520"}
Sep 16 17:25:51.520645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:51.520614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfjgf"]
Sep 16 17:25:51.524097 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:25:51.524072    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4de0343_408d_44a6_8319_b4d446c75750.slice/crio-c43f4f606a9761e39ebfe91c4a245358770b88a66f16be063e8f0b1d362f97eb WatchSource:0}: Error finding container c43f4f606a9761e39ebfe91c4a245358770b88a66f16be063e8f0b1d362f97eb: Status 404 returned error can't find the container with id c43f4f606a9761e39ebfe91c4a245358770b88a66f16be063e8f0b1d362f97eb
Sep 16 17:25:52.362798 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:52.362755    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfjgf" event={"ID":"a4de0343-408d-44a6-8319-b4d446c75750","Type":"ContainerStarted","Data":"c43f4f606a9761e39ebfe91c4a245358770b88a66f16be063e8f0b1d362f97eb"}
Sep 16 17:25:56.376043 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:56.376008    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5sjzg" event={"ID":"0640143a-e0a6-4db6-85be-93f4cf337db9","Type":"ContainerStarted","Data":"fbc26d329ba6abbc53fe42ea15a674435bc6bcb52ab62eec2c1215ec796340f5"}
Sep 16 17:25:56.377499 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:56.377477    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfjgf" event={"ID":"a4de0343-408d-44a6-8319-b4d446c75750","Type":"ContainerStarted","Data":"2e4b27e961a0a6f6be631c94b05020eaafce56ea524f853dbcb18a452743e78e"}
Sep 16 17:25:56.377634 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:56.377620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:25:56.391791 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:56.391751    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5sjzg" podStartSLOduration=66.918798675 podStartE2EDuration="1m11.391736404s" totalImagesPullingTime="4.472937729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:56.391139714 +0000 UTC m=+71.800256965" watchObservedRunningTime="2026-09-16 17:25:56.391736404 +0000 UTC m=+71.800853651"
Sep 16 17:25:56.406061 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:25:56.406017    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mfjgf" podStartSLOduration=67.167304938 podStartE2EDuration="1m11.406006897s" totalImagesPullingTime="4.238701959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:25:56.405952118 +0000 UTC m=+71.815069367" watchObservedRunningTime="2026-09-16 17:25:56.406006897 +0000 UTC m=+71.815124145"
Sep 16 17:26:22.275687 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:26:22.275653    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:26:22.276045 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:26:22.275753    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 17:26:22.276045 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:26:22.275814    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls podName:18dd06e3-0da0-46da-b08e-d45b2396b86b nodeName:}" failed. No retries permitted until 2026-09-16 17:27:26.275800754 +0000 UTC m=+161.684917979 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls") pod "dns-default-xpk25" (UID: "18dd06e3-0da0-46da-b08e-d45b2396b86b") : secret "dns-default-metrics-tls" not found
Sep 16 17:26:27.382397 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:26:27.382364    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mfjgf"
Sep 16 17:26:54.801445 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:26:54.801410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:26:54.801931 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:26:54.801520    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 17:26:54.801931 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:26:54.801641    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs podName:4304ee43-101e-44b9-bfc7-87b2e0a9273f nodeName:}" failed. No retries permitted until 2026-09-16 17:28:56.80162408 +0000 UTC m=+252.210741309 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs") pod "network-metrics-daemon-bfp6l" (UID: "4304ee43-101e-44b9-bfc7-87b2e0a9273f") : secret "metrics-daemon-secret" not found
Sep 16 17:27:07.432736 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.432699    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qbdw9"]
Sep 16 17:27:07.434515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.434498    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.436919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.436891    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 17:27:07.437149 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.437130    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 17:27:07.437437 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.437421    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hmpn4\""
Sep 16 17:27:07.437574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.437560    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 17:27:07.437973 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.437956    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 17:27:07.458066 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.458047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbdw9"]
Sep 16 17:27:07.586716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.586689    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9ql\" (UniqueName: \"kubernetes.io/projected/e93d211c-97dd-4665-b91d-f5df364301f3-kube-api-access-ps9ql\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.586716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.586721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.586903 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.586747    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e93d211c-97dd-4665-b91d-f5df364301f3-crio-socket\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.586903 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.586767    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93d211c-97dd-4665-b91d-f5df364301f3-data-volume\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.586903 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.586802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687439 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687381    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9ql\" (UniqueName: \"kubernetes.io/projected/e93d211c-97dd-4665-b91d-f5df364301f3-kube-api-access-ps9ql\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687439 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687413    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e93d211c-97dd-4665-b91d-f5df364301f3-crio-socket\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93d211c-97dd-4665-b91d-f5df364301f3-data-volume\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687547 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:07.687521    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:07.687547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e93d211c-97dd-4665-b91d-f5df364301f3-crio-socket\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.687759 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:07.687576    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:07.687759 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:07.687625    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:08.187579898 +0000 UTC m=+143.596697128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:07.687759 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:07.687647    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:08.187631605 +0000 UTC m=+143.596748845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:07.687872 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.687762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e93d211c-97dd-4665-b91d-f5df364301f3-data-volume\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:07.699960 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:07.699939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9ql\" (UniqueName: \"kubernetes.io/projected/e93d211c-97dd-4665-b91d-f5df364301f3-kube-api-access-ps9ql\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:08.191679 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:08.191640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:08.191847 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:08.191724    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:08.191847 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:08.191802    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:08.191917 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:08.191859    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.19184505 +0000 UTC m=+144.600962275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:08.191917 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:08.191802    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:08.191988 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:08.191938    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:09.191925253 +0000 UTC m=+144.601042492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.200047 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:09.200020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:09.200412 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:09.200095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:09.200412 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:09.200164    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.200412 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:09.200226    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:11.200206266 +0000 UTC m=+146.609323495 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:09.200412 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:09.200164    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:09.200412 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:09.200305    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:11.200293047 +0000 UTC m=+146.609410288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:10.321370 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:10.321341    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q426q_3a3f53ed-df0f-4f40-94c1-0fdacaa54f70/dns-node-resolver/0.log"
Sep 16 17:27:11.216174 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:11.216141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:11.216357 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:11.216186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:11.216357 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:11.216288    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:11.216357 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:11.216303    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:11.216357 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:11.216355    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.216342586 +0000 UTC m=+150.625459811 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:11.216515 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:11.216370    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:15.216363234 +0000 UTC m=+150.625480459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:11.316067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:11.316044    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w5vt2_7c76ab09-5936-4425-94f8-dd03433ecfa9/node-ca/0.log"
Sep 16 17:27:12.123603 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:12.123560    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qlnrn_41b04f03-bcb0-49ad-ac5d-a4cf82768494/serve-healthcheck-canary/0.log"
Sep 16 17:27:15.249882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:15.249848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:15.250242 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:15.249894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:15.250242 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:15.249960    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 17:27:15.250242 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:15.250015    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:15.250242 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:15.250017    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:23.250002856 +0000 UTC m=+158.659120082 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : configmap "kube-rbac-proxy" not found
Sep 16 17:27:15.250242 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:15.250075    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls podName:e93d211c-97dd-4665-b91d-f5df364301f3 nodeName:}" failed. No retries permitted until 2026-09-16 17:27:23.250059337 +0000 UTC m=+158.659176563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qbdw9" (UID: "e93d211c-97dd-4665-b91d-f5df364301f3") : secret "insights-runtime-extractor-tls" not found
Sep 16 17:27:21.501518 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:21.501475    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xpk25" podUID="18dd06e3-0da0-46da-b08e-d45b2396b86b"
Sep 16 17:27:21.622016 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:21.621988    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xpk25"
Sep 16 17:27:23.130668 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:23.130633    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-bfp6l" podUID="4304ee43-101e-44b9-bfc7-87b2e0a9273f"
Sep 16 17:27:23.312197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.312165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:23.312346 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.312213    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:23.312817 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.312794    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e93d211c-97dd-4665-b91d-f5df364301f3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:23.314473 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.314455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e93d211c-97dd-4665-b91d-f5df364301f3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qbdw9\" (UID: \"e93d211c-97dd-4665-b91d-f5df364301f3\") " pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:23.342706 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.342666    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qbdw9"
Sep 16 17:27:23.611611 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.611570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qbdw9"]
Sep 16 17:27:23.615114 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:23.615087    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93d211c_97dd_4665_b91d_f5df364301f3.slice/crio-c5076985268a57b6ea90d736858d1aba8ffe2bd62eaf3def8c7f539c05ce6175 WatchSource:0}: Error finding container c5076985268a57b6ea90d736858d1aba8ffe2bd62eaf3def8c7f539c05ce6175: Status 404 returned error can't find the container with id c5076985268a57b6ea90d736858d1aba8ffe2bd62eaf3def8c7f539c05ce6175
Sep 16 17:27:23.628759 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:23.628709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbdw9" event={"ID":"e93d211c-97dd-4665-b91d-f5df364301f3","Type":"ContainerStarted","Data":"c5076985268a57b6ea90d736858d1aba8ffe2bd62eaf3def8c7f539c05ce6175"}
Sep 16 17:27:24.635076 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:24.635038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbdw9" event={"ID":"e93d211c-97dd-4665-b91d-f5df364301f3","Type":"ContainerStarted","Data":"d54131d6e32f6d811ce90fe32493da524f39b24d9e14a8aed992c43254a9d022"}
Sep 16 17:27:24.635450 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:24.635097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbdw9" event={"ID":"e93d211c-97dd-4665-b91d-f5df364301f3","Type":"ContainerStarted","Data":"0300f36e2efad5231917d9c209e5da0980cab4f573140e1c64745f658bf84fa2"}
Sep 16 17:27:25.640072 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:25.640000    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qbdw9" event={"ID":"e93d211c-97dd-4665-b91d-f5df364301f3","Type":"ContainerStarted","Data":"51f566e91482940283d0524ac8e3ed15ad0619102aced18dea08de3e651eb6e4"}
Sep 16 17:27:25.662139 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:25.662098    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qbdw9" podStartSLOduration=16.977072874 podStartE2EDuration="18.662084622s" totalImagesPullingTime="1.685011748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:25.661896292 +0000 UTC m=+161.071013540" watchObservedRunningTime="2026-09-16 17:27:25.662084622 +0000 UTC m=+161.071201869"
Sep 16 17:27:26.334623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:26.334565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:27:26.336820 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:26.336800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/18dd06e3-0da0-46da-b08e-d45b2396b86b-metrics-tls\") pod \"dns-default-xpk25\" (UID: \"18dd06e3-0da0-46da-b08e-d45b2396b86b\") " pod="openshift-dns/dns-default-xpk25"
Sep 16 17:27:26.424933 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:26.424910    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qld9l\""
Sep 16 17:27:26.432898 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:26.432867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xpk25"
Sep 16 17:27:26.742434 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:26.742408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xpk25"]
Sep 16 17:27:26.745323 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:26.745301    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dd06e3_0da0_46da_b08e_d45b2396b86b.slice/crio-cd994aa128c65e993161aadc7961ef6009d661c7d21912ea7639fd096806e217 WatchSource:0}: Error finding container cd994aa128c65e993161aadc7961ef6009d661c7d21912ea7639fd096806e217: Status 404 returned error can't find the container with id cd994aa128c65e993161aadc7961ef6009d661c7d21912ea7639fd096806e217
Sep 16 17:27:27.648091 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:27.648050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xpk25" event={"ID":"18dd06e3-0da0-46da-b08e-d45b2396b86b","Type":"ContainerStarted","Data":"cd994aa128c65e993161aadc7961ef6009d661c7d21912ea7639fd096806e217"}
Sep 16 17:27:28.653608 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:28.653556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xpk25" event={"ID":"18dd06e3-0da0-46da-b08e-d45b2396b86b","Type":"ContainerStarted","Data":"fab5b5a63e11ed4066b77725f79cf504d1c4e3061e08ee3df15ae61e7f320d26"}
Sep 16 17:27:28.654103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:28.653653    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xpk25"
Sep 16 17:27:28.654103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:28.653679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xpk25" event={"ID":"18dd06e3-0da0-46da-b08e-d45b2396b86b","Type":"ContainerStarted","Data":"c24d4bdcf39a3d4c7603f8716b4fb4e2a99ad0dba4b1ec57aca6603d1a2d2666"}
Sep 16 17:27:28.687375 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:28.687328    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xpk25" podStartSLOduration=129.294805933 podStartE2EDuration="2m10.687313105s" totalImagesPullingTime="1.392507172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:25:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:28.685400362 +0000 UTC m=+164.094517610" watchObservedRunningTime="2026-09-16 17:27:28.687313105 +0000 UTC m=+164.096430411"
Sep 16 17:27:31.847186 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:31.847153    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"]
Sep 16 17:27:31.849411 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:31.849395    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:31.854549 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:31.854529    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k65xg\""
Sep 16 17:27:31.854651 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:31.854535    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 17:27:31.887392 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:31.887368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"]
Sep 16 17:27:31.974786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:31.974758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fff31bbe-8f93-4523-b374-b7e691f67780-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cbpg2\" (UID: \"fff31bbe-8f93-4523-b374-b7e691f67780\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:32.075337 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:32.075309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fff31bbe-8f93-4523-b374-b7e691f67780-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cbpg2\" (UID: \"fff31bbe-8f93-4523-b374-b7e691f67780\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:32.077717 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:32.077698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fff31bbe-8f93-4523-b374-b7e691f67780-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-cbpg2\" (UID: \"fff31bbe-8f93-4523-b374-b7e691f67780\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:32.158484 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:32.158404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:32.435366 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:32.435294    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"]
Sep 16 17:27:32.441148 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:32.441107    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff31bbe_8f93_4523_b374_b7e691f67780.slice/crio-6e738a0fd79ff624cd99bb4b85e765eb717db0fb0600979d6d627ac78636a1a0 WatchSource:0}: Error finding container 6e738a0fd79ff624cd99bb4b85e765eb717db0fb0600979d6d627ac78636a1a0: Status 404 returned error can't find the container with id 6e738a0fd79ff624cd99bb4b85e765eb717db0fb0600979d6d627ac78636a1a0
Sep 16 17:27:32.668339 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:32.668306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2" event={"ID":"fff31bbe-8f93-4523-b374-b7e691f67780","Type":"ContainerStarted","Data":"6e738a0fd79ff624cd99bb4b85e765eb717db0fb0600979d6d627ac78636a1a0"}
Sep 16 17:27:33.673913 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:33.673824    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2" event={"ID":"fff31bbe-8f93-4523-b374-b7e691f67780","Type":"ContainerStarted","Data":"02c3846418f511272ff900672df71ad23e56948f96c7f9bc4d97f371a1d4615e"}
Sep 16 17:27:33.674270 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:33.674043    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:33.678616 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:33.678577    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2"
Sep 16 17:27:33.700113 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:33.700070    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-cbpg2" podStartSLOduration=1.7627625660000001 podStartE2EDuration="2.700056125s" totalImagesPullingTime="937.293559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:33.699172281 +0000 UTC m=+169.108289529" watchObservedRunningTime="2026-09-16 17:27:33.700056125 +0000 UTC m=+169.109173412"
Sep 16 17:27:34.112439 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.112403    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:27:34.780800 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.780769    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-586c9cb5bf-t7f89"]
Sep 16 17:27:34.782827 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.782800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.794681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.794662    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 17:27:34.795021 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.795007    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 17:27:34.795667 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:34.795646    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 17:27:34.795729 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.795674    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 17:27:34.795805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.795789    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-djvkl\""
Sep 16 17:27:34.796172 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.796158    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 17:27:34.796764 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.796752    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 17:27:34.797012 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:27:34.796993    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"console-serving-cert\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"console-serving-cert\"" type="*v1.Secret"
Sep 16 17:27:34.827829 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.827808    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586c9cb5bf-t7f89"]
Sep 16 17:27:34.894789 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.894758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-serving-cert\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.894789 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.894788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-console-config\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.894970 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.894809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-service-ca\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.894970 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.894830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-oauth-config\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.894970 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.894943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7s8h\" (UniqueName: \"kubernetes.io/projected/b4913016-d411-4760-b16d-dde5946a416b-kube-api-access-r7s8h\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.895083 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.894976    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-oauth-serving-cert\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.995716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.995676    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7s8h\" (UniqueName: \"kubernetes.io/projected/b4913016-d411-4760-b16d-dde5946a416b-kube-api-access-r7s8h\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.995887 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.995734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-oauth-serving-cert\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.995887 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.995759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-serving-cert\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.995887 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.995779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-console-config\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.995887 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.995803    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-service-ca\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.995887 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.995832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-oauth-config\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.996505 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.996483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-oauth-serving-cert\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.996667 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.996647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-service-ca\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.996706 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.996647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-console-config\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:34.998250 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:34.998232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-oauth-config\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:35.735045 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:35.735010    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 17:27:35.738321 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:35.738293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7s8h\" (UniqueName: \"kubernetes.io/projected/b4913016-d411-4760-b16d-dde5946a416b-kube-api-access-r7s8h\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:35.970538 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:35.970508    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 17:27:35.978763 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:35.978735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-serving-cert\") pod \"console-586c9cb5bf-t7f89\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") " pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:35.991753 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:35.991702    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:36.264465 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:36.264390    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-586c9cb5bf-t7f89"]
Sep 16 17:27:36.267238 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:36.267209    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4913016_d411_4760_b16d_dde5946a416b.slice/crio-07d2aebc3ad6a99834636e23d6318e1af95efc856b7f0019db81d52c7400bde5 WatchSource:0}: Error finding container 07d2aebc3ad6a99834636e23d6318e1af95efc856b7f0019db81d52c7400bde5: Status 404 returned error can't find the container with id 07d2aebc3ad6a99834636e23d6318e1af95efc856b7f0019db81d52c7400bde5
Sep 16 17:27:36.684913 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:36.684835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cb5bf-t7f89" event={"ID":"b4913016-d411-4760-b16d-dde5946a416b","Type":"ContainerStarted","Data":"07d2aebc3ad6a99834636e23d6318e1af95efc856b7f0019db81d52c7400bde5"}
Sep 16 17:27:38.661545 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:38.661328    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xpk25"
Sep 16 17:27:39.449442 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.449412    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"]
Sep 16 17:27:39.451409 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.451390    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lw8hl"]
Sep 16 17:27:39.451520 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.451503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.453564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.453541    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.455582 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.455562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 16 17:27:39.455765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.455750    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 16 17:27:39.455862 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.455845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 17:27:39.455919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.455867    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 17:27:39.455919 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.455910    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 17:27:39.470598 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.470568    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 17:27:39.470816 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.470799    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 17:27:39.470891 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.470845    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gvkfk\""
Sep 16 17:27:39.470954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.470896    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 17:27:39.471507 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.471490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fgdx2\""
Sep 16 17:27:39.481319 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.481296    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"]
Sep 16 17:27:39.523780 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523756    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.523875 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-root\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.523875 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4ff827-ef40-479c-a127-d67bba6be3e7-metrics-client-ca\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.523875 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.523875 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523857    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-sys\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.524009 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfs5g\" (UniqueName: \"kubernetes.io/projected/ce4ff827-ef40-479c-a127-d67bba6be3e7-kube-api-access-kfs5g\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.524009 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523941    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-tls\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.524009 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.523979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.524103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.524009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-wtmp\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.524103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.524027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.524103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.524048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6f5r\" (UniqueName: \"kubernetes.io/projected/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-kube-api-access-z6f5r\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.524103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.524072    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-textfile\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.524334 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.524103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625015 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.624983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-root\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4ff827-ef40-479c-a127-d67bba6be3e7-metrics-client-ca\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625063    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-sys\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-root\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfs5g\" (UniqueName: \"kubernetes.io/projected/ce4ff827-ef40-479c-a127-d67bba6be3e7-kube-api-access-kfs5g\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625162    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-tls\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-wtmp\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625223    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6f5r\" (UniqueName: \"kubernetes.io/projected/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-kube-api-access-z6f5r\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625268    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-textfile\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-sys\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625900 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce4ff827-ef40-479c-a127-d67bba6be3e7-metrics-client-ca\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625952 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-wtmp\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.625952 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625930    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.626054 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.625995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.626109 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.626082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-textfile\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.627654 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.627610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.627654 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.627654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.627825 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.627795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.628027 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.628008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ce4ff827-ef40-479c-a127-d67bba6be3e7-node-exporter-tls\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.645529 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.645504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6f5r\" (UniqueName: \"kubernetes.io/projected/8a82b3b4-164d-4b02-ad13-a0b6972f33bf-kube-api-access-z6f5r\") pod \"openshift-state-metrics-67dcf8c979-nfc4b\" (UID: \"8a82b3b4-164d-4b02-ad13-a0b6972f33bf\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.649155 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.649136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfs5g\" (UniqueName: \"kubernetes.io/projected/ce4ff827-ef40-479c-a127-d67bba6be3e7-kube-api-access-kfs5g\") pod \"node-exporter-lw8hl\" (UID: \"ce4ff827-ef40-479c-a127-d67bba6be3e7\") " pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.701436 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.701365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cb5bf-t7f89" event={"ID":"b4913016-d411-4760-b16d-dde5946a416b","Type":"ContainerStarted","Data":"106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2"}
Sep 16 17:27:39.737909 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.737866    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-586c9cb5bf-t7f89" podStartSLOduration=3.061450834 podStartE2EDuration="5.737853362s" totalImagesPullingTime="2.676402528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:39.736953116 +0000 UTC m=+175.146070364" watchObservedRunningTime="2026-09-16 17:27:39.737853362 +0000 UTC m=+175.146970611"
Sep 16 17:27:39.761154 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.761133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"
Sep 16 17:27:39.765769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:39.765748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lw8hl"
Sep 16 17:27:39.775064 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:39.775032    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4ff827_ef40_479c_a127_d67bba6be3e7.slice/crio-abc67c04f12f7cf77165d99788ae94b827a7d77598eb6496113206d4d7b42fb5 WatchSource:0}: Error finding container abc67c04f12f7cf77165d99788ae94b827a7d77598eb6496113206d4d7b42fb5: Status 404 returned error can't find the container with id abc67c04f12f7cf77165d99788ae94b827a7d77598eb6496113206d4d7b42fb5
Sep 16 17:27:40.055850 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:40.055806    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b"]
Sep 16 17:27:40.058460 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:40.058435    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a82b3b4_164d_4b02_ad13_a0b6972f33bf.slice/crio-56740eade5843ed7172ee9ddecf1f91a62f6df6cdc17fb7e374c4d30904601ec WatchSource:0}: Error finding container 56740eade5843ed7172ee9ddecf1f91a62f6df6cdc17fb7e374c4d30904601ec: Status 404 returned error can't find the container with id 56740eade5843ed7172ee9ddecf1f91a62f6df6cdc17fb7e374c4d30904601ec
Sep 16 17:27:40.707977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:40.707944    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b" event={"ID":"8a82b3b4-164d-4b02-ad13-a0b6972f33bf","Type":"ContainerStarted","Data":"3c4f8c27d61509c9e4e3197d2ff436239bfd70b402ff3c9bf1af1f67b931fc1c"}
Sep 16 17:27:40.708389 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:40.708022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b" event={"ID":"8a82b3b4-164d-4b02-ad13-a0b6972f33bf","Type":"ContainerStarted","Data":"011c47c9332b8451cd05a3e72da110136daa1c06553b9b82dd9387550ade30c1"}
Sep 16 17:27:40.708389 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:40.708051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b" event={"ID":"8a82b3b4-164d-4b02-ad13-a0b6972f33bf","Type":"ContainerStarted","Data":"56740eade5843ed7172ee9ddecf1f91a62f6df6cdc17fb7e374c4d30904601ec"}
Sep 16 17:27:40.709522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:40.709499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw8hl" event={"ID":"ce4ff827-ef40-479c-a127-d67bba6be3e7","Type":"ContainerStarted","Data":"41874976d36a50a8433c2f9a6a91962333b0c456afdf17f870d0818ae48b50fb"}
Sep 16 17:27:40.709643 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:40.709552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw8hl" event={"ID":"ce4ff827-ef40-479c-a127-d67bba6be3e7","Type":"ContainerStarted","Data":"abc67c04f12f7cf77165d99788ae94b827a7d77598eb6496113206d4d7b42fb5"}
Sep 16 17:27:41.714811 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:41.714776    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b" event={"ID":"8a82b3b4-164d-4b02-ad13-a0b6972f33bf","Type":"ContainerStarted","Data":"5ffe500dbb8d7a2c92247bb86cd1d651ed2a29501774b474fdb405630c058c83"}
Sep 16 17:27:41.716175 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:41.716154    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce4ff827-ef40-479c-a127-d67bba6be3e7" containerID="41874976d36a50a8433c2f9a6a91962333b0c456afdf17f870d0818ae48b50fb" exitCode=0
Sep 16 17:27:41.716251 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:41.716201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw8hl" event={"ID":"ce4ff827-ef40-479c-a127-d67bba6be3e7","Type":"ContainerDied","Data":"41874976d36a50a8433c2f9a6a91962333b0c456afdf17f870d0818ae48b50fb"}
Sep 16 17:27:41.732393 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:41.732346    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-nfc4b" podStartSLOduration=1.5541148150000001 podStartE2EDuration="2.732326113s" totalImagesPullingTime="1.178211298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:41.73160757 +0000 UTC m=+177.140724818" watchObservedRunningTime="2026-09-16 17:27:41.732326113 +0000 UTC m=+177.141443362"
Sep 16 17:27:42.722461 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:42.722428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw8hl" event={"ID":"ce4ff827-ef40-479c-a127-d67bba6be3e7","Type":"ContainerStarted","Data":"d85ee110afd7128062acd59edf7db9a9cd07e3bd76d4b4f06ad4604da66f3959"}
Sep 16 17:27:42.722847 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:42.722486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lw8hl" event={"ID":"ce4ff827-ef40-479c-a127-d67bba6be3e7","Type":"ContainerStarted","Data":"c2b06b81a717a293f04f271e89cbb3ff57cb4cd0dd254331c54e90259a8f6211"}
Sep 16 17:27:42.741391 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:42.741334    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lw8hl" podStartSLOduration=2.877467964 podStartE2EDuration="3.741319159s" totalImagesPullingTime="801.088398ms" totalInitContainerRuntime="62.762797ms" isStatefulPod=true podCreationTimestamp="2026-09-16 17:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:42.740837396 +0000 UTC m=+178.149954645" watchObservedRunningTime="2026-09-16 17:27:42.741319159 +0000 UTC m=+178.150436438"
Sep 16 17:27:43.268111 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.268057    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-898987c6b-cb8wh"]
Sep 16 17:27:43.270480 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.270466    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.272521 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.272486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 17:27:43.272521 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.272489    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-p578l\""
Sep 16 17:27:43.272830 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.272807    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 17:27:43.272923 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.272850    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 17:27:43.273133 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.273111    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-civdiua1u0leq\""
Sep 16 17:27:43.273233 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.273115    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 17:27:43.273233 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.273151    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 17:27:43.288969 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.288946    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-898987c6b-cb8wh"]
Sep 16 17:27:43.357673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357679    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klnp\" (UniqueName: \"kubernetes.io/projected/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-kube-api-access-4klnp\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357859 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-grpc-tls\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357859 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357806    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-metrics-client-ca\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357859 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357984 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-tls\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357984 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.357984 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.357937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459192 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459192 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459192    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-tls\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klnp\" (UniqueName: \"kubernetes.io/projected/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-kube-api-access-4klnp\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-grpc-tls\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.459692 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.459519    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-metrics-client-ca\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.460306 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.460278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-metrics-client-ca\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.461915 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.461893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-tls\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.462117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.462096    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.462258 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.462238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.462319 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.462259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.462371 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.462316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.462681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.462662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-secret-grpc-tls\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.466993 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.466971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klnp\" (UniqueName: \"kubernetes.io/projected/b6474a2f-907a-4ff2-bcf9-77f63789d1a1-kube-api-access-4klnp\") pod \"thanos-querier-898987c6b-cb8wh\" (UID: \"b6474a2f-907a-4ff2-bcf9-77f63789d1a1\") " pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.579937 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.579854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:43.862832 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:43.862760    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-898987c6b-cb8wh"]
Sep 16 17:27:43.866054 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:43.866027    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6474a2f_907a_4ff2_bcf9_77f63789d1a1.slice/crio-a56c5c6fa7d42e051d4b08afadc59e570b2bb8d8f870625df76a5db918c01482 WatchSource:0}: Error finding container a56c5c6fa7d42e051d4b08afadc59e570b2bb8d8f870625df76a5db918c01482: Status 404 returned error can't find the container with id a56c5c6fa7d42e051d4b08afadc59e570b2bb8d8f870625df76a5db918c01482
Sep 16 17:27:44.730320 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:44.730282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"a56c5c6fa7d42e051d4b08afadc59e570b2bb8d8f870625df76a5db918c01482"}
Sep 16 17:27:45.736022 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:45.735976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"c4a0060aa4a4de926f2fe165beadbf6a17cb7b9aedcb16d038cf67f96d9f0980"}
Sep 16 17:27:45.736381 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:45.736075    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"d913ebe1cf4dc8ea98bc2a8c3b0db156c4cf25d5f775d314fce08bf38e15f864"}
Sep 16 17:27:45.736381 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:45.736102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"f8160f16e23697eae4f6305de2e3265f8f6462f08fdbf55b6284da0b34eb0e8b"}
Sep 16 17:27:45.992637 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:45.992527    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:45.992637 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:45.992620    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:45.997311 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:45.997290    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:46.250643 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.250552    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7695756959-bk6dn"]
Sep 16 17:27:46.253125 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.253094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.262957 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.262930    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 17:27:46.284544 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-oauth-serving-cert\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.284684 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-service-ca\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.284684 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-oauth-config\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.284785 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-serving-cert\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.284834 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-console-config\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.284871 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnjk\" (UniqueName: \"kubernetes.io/projected/16913a21-4551-4fcc-bcd7-afba75ad821c-kube-api-access-2fnjk\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.284925 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.284913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-trusted-ca-bundle\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.317947 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.317926    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7695756959-bk6dn"]
Sep 16 17:27:46.385203 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-console-config\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.385301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnjk\" (UniqueName: \"kubernetes.io/projected/16913a21-4551-4fcc-bcd7-afba75ad821c-kube-api-access-2fnjk\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.385301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-trusted-ca-bundle\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.385301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-oauth-serving-cert\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.385301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-service-ca\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.385467 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-oauth-config\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.385467 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-serving-cert\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.386026 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.385998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-oauth-serving-cert\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.386140 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.386008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-console-config\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.386199 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.386138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-trusted-ca-bundle\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.386199 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.386159    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-service-ca\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.388029 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.387999    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-serving-cert\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.388107 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.388011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-oauth-config\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.394371 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.394343    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnjk\" (UniqueName: \"kubernetes.io/projected/16913a21-4551-4fcc-bcd7-afba75ad821c-kube-api-access-2fnjk\") pod \"console-7695756959-bk6dn\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") " pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.564393 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.564318    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:46.742196 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.742159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"38273e3b898add68b04ef18580e5d610b57640aa4b4c27edc513b8f29b2707b4"}
Sep 16 17:27:46.742548 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.742236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"cc313b451ed666a10a11ef61943cb465300c9453555d7cd7eebcee60b1dd4863"}
Sep 16 17:27:46.742548 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.742264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" event={"ID":"b6474a2f-907a-4ff2-bcf9-77f63789d1a1","Type":"ContainerStarted","Data":"af8dc450dc9eaa5afc808d0f7ec0f3b3674394745b794e78626866135f8a0eb3"}
Sep 16 17:27:46.746077 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.746049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:27:46.807687 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.807646    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh" podStartSLOduration=1.526144474 podStartE2EDuration="3.807630984s" totalImagesPullingTime="2.28148651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:46.781744153 +0000 UTC m=+182.190861400" watchObservedRunningTime="2026-09-16 17:27:46.807630984 +0000 UTC m=+182.216748233"
Sep 16 17:27:46.844840 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:46.844671    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7695756959-bk6dn"]
Sep 16 17:27:46.847371 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:27:46.847343    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16913a21_4551_4fcc_bcd7_afba75ad821c.slice/crio-089996496a384206eb3e3f289b1b33b29f7f305c719faba6f5e9dc53bd75ba07 WatchSource:0}: Error finding container 089996496a384206eb3e3f289b1b33b29f7f305c719faba6f5e9dc53bd75ba07: Status 404 returned error can't find the container with id 089996496a384206eb3e3f289b1b33b29f7f305c719faba6f5e9dc53bd75ba07
Sep 16 17:27:47.747186 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:47.747152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695756959-bk6dn" event={"ID":"16913a21-4551-4fcc-bcd7-afba75ad821c","Type":"ContainerStarted","Data":"765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66"}
Sep 16 17:27:47.747569 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:47.747201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695756959-bk6dn" event={"ID":"16913a21-4551-4fcc-bcd7-afba75ad821c","Type":"ContainerStarted","Data":"089996496a384206eb3e3f289b1b33b29f7f305c719faba6f5e9dc53bd75ba07"}
Sep 16 17:27:47.747569 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:47.747547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:47.774516 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:47.774441    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7695756959-bk6dn" podStartSLOduration=1.774423449 podStartE2EDuration="1.774423449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:27:47.768530024 +0000 UTC m=+183.177647272" watchObservedRunningTime="2026-09-16 17:27:47.774423449 +0000 UTC m=+183.183540696"
Sep 16 17:27:53.756633 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:53.756605    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-898987c6b-cb8wh"
Sep 16 17:27:56.565048 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:56.565000    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:56.565516 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:56.565069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:56.569712 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:56.569688    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:56.779126 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:56.779099    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:27:56.881155 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:27:56.881086    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586c9cb5bf-t7f89"]
Sep 16 17:28:19.042181 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:19.042135    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" podUID="79f9fecd-7d6a-4411-a444-92d747320af6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 17:28:21.899937 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:21.899876    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-586c9cb5bf-t7f89" podUID="b4913016-d411-4760-b16d-dde5946a416b" containerName="console" containerID="cri-o://106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2" gracePeriod=15
Sep 16 17:28:22.159402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.159349    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:28:22.250562 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.250535    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-oauth-serving-cert\") pod \"b4913016-d411-4760-b16d-dde5946a416b\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") "
Sep 16 17:28:22.250705 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.250577    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7s8h\" (UniqueName: \"kubernetes.io/projected/b4913016-d411-4760-b16d-dde5946a416b-kube-api-access-r7s8h\") pod \"b4913016-d411-4760-b16d-dde5946a416b\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") "
Sep 16 17:28:22.250705 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.250647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-serving-cert\") pod \"b4913016-d411-4760-b16d-dde5946a416b\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") "
Sep 16 17:28:22.250705 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.250674    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-oauth-config\") pod \"b4913016-d411-4760-b16d-dde5946a416b\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") "
Sep 16 17:28:22.250832 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.250713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-console-config\") pod \"b4913016-d411-4760-b16d-dde5946a416b\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") "
Sep 16 17:28:22.250832 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.250737    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-service-ca\") pod \"b4913016-d411-4760-b16d-dde5946a416b\" (UID: \"b4913016-d411-4760-b16d-dde5946a416b\") "
Sep 16 17:28:22.251257 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.251209    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-service-ca" (OuterVolumeSpecName: "service-ca") pod "b4913016-d411-4760-b16d-dde5946a416b" (UID: "b4913016-d411-4760-b16d-dde5946a416b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:22.251257 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.251229    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-console-config" (OuterVolumeSpecName: "console-config") pod "b4913016-d411-4760-b16d-dde5946a416b" (UID: "b4913016-d411-4760-b16d-dde5946a416b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:22.251420 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.251279    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b4913016-d411-4760-b16d-dde5946a416b" (UID: "b4913016-d411-4760-b16d-dde5946a416b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:28:22.252830 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.252804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4913016-d411-4760-b16d-dde5946a416b-kube-api-access-r7s8h" (OuterVolumeSpecName: "kube-api-access-r7s8h") pod "b4913016-d411-4760-b16d-dde5946a416b" (UID: "b4913016-d411-4760-b16d-dde5946a416b"). InnerVolumeSpecName "kube-api-access-r7s8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:28:22.252972 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.252948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b4913016-d411-4760-b16d-dde5946a416b" (UID: "b4913016-d411-4760-b16d-dde5946a416b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:22.253206 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.253179    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b4913016-d411-4760-b16d-dde5946a416b" (UID: "b4913016-d411-4760-b16d-dde5946a416b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:28:22.352084 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.352054    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-oauth-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:28:22.352084 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.352078    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-console-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:28:22.352084 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.352087    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-service-ca\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:28:22.352267 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.352095    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b4913016-d411-4760-b16d-dde5946a416b-oauth-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:28:22.352267 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.352104    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7s8h\" (UniqueName: \"kubernetes.io/projected/b4913016-d411-4760-b16d-dde5946a416b-kube-api-access-r7s8h\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:28:22.352267 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.352112    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4913016-d411-4760-b16d-dde5946a416b-console-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:28:22.863504 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.863473    2560 generic.go:356] "Generic (PLEG): container finished" podID="b4913016-d411-4760-b16d-dde5946a416b" containerID="106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2" exitCode=0
Sep 16 17:28:22.863700 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.863519    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-586c9cb5bf-t7f89"
Sep 16 17:28:22.863700 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.863525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cb5bf-t7f89" event={"ID":"b4913016-d411-4760-b16d-dde5946a416b","Type":"ContainerDied","Data":"106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2"}
Sep 16 17:28:22.863700 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.863568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-586c9cb5bf-t7f89" event={"ID":"b4913016-d411-4760-b16d-dde5946a416b","Type":"ContainerDied","Data":"07d2aebc3ad6a99834636e23d6318e1af95efc856b7f0019db81d52c7400bde5"}
Sep 16 17:28:22.863700 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.863609    2560 scope.go:117] "RemoveContainer" containerID="106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2"
Sep 16 17:28:22.871800 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.871785    2560 scope.go:117] "RemoveContainer" containerID="106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2"
Sep 16 17:28:22.872077 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:28:22.872054    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2\": container with ID starting with 106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2 not found: ID does not exist" containerID="106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2"
Sep 16 17:28:22.872169 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.872081    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2"} err="failed to get container status \"106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2\": rpc error: code = NotFound desc = could not find container \"106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2\": container with ID starting with 106626ae75e0067606617a91ee02406dec8081646ad20d24d52c9a62f31105c2 not found: ID does not exist"
Sep 16 17:28:22.886670 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.885153    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-586c9cb5bf-t7f89"]
Sep 16 17:28:22.890184 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:22.890163    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-586c9cb5bf-t7f89"]
Sep 16 17:28:23.116536 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:23.116460    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4913016-d411-4760-b16d-dde5946a416b" path="/var/lib/kubelet/pods/b4913016-d411-4760-b16d-dde5946a416b/volumes"
Sep 16 17:28:29.042320 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:29.042285    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" podUID="79f9fecd-7d6a-4411-a444-92d747320af6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 17:28:39.042764 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.042727    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" podUID="79f9fecd-7d6a-4411-a444-92d747320af6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 17:28:39.043137 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.042806    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9"
Sep 16 17:28:39.043254 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.043239    2560 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"948e62e657f629d2c24de6de03d58b1f9e09e6460c9b3e5d84b75cc24890d59f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 17:28:39.043296 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.043274    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" podUID="79f9fecd-7d6a-4411-a444-92d747320af6" containerName="service-proxy" containerID="cri-o://948e62e657f629d2c24de6de03d58b1f9e09e6460c9b3e5d84b75cc24890d59f" gracePeriod=30
Sep 16 17:28:39.926905 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.926874    2560 generic.go:356] "Generic (PLEG): container finished" podID="79f9fecd-7d6a-4411-a444-92d747320af6" containerID="948e62e657f629d2c24de6de03d58b1f9e09e6460c9b3e5d84b75cc24890d59f" exitCode=2
Sep 16 17:28:39.927063 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.926943    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" event={"ID":"79f9fecd-7d6a-4411-a444-92d747320af6","Type":"ContainerDied","Data":"948e62e657f629d2c24de6de03d58b1f9e09e6460c9b3e5d84b75cc24890d59f"}
Sep 16 17:28:39.927063 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:39.926987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57f46664dd-ml5n9" event={"ID":"79f9fecd-7d6a-4411-a444-92d747320af6","Type":"ContainerStarted","Data":"620a5530510f30afd54846f394bdc4f3a10680f9aa45edce6e7ee68302f2fbb4"}
Sep 16 17:28:56.819562 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:56.819520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:28:56.821845 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:56.821827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4304ee43-101e-44b9-bfc7-87b2e0a9273f-metrics-certs\") pod \"network-metrics-daemon-bfp6l\" (UID: \"4304ee43-101e-44b9-bfc7-87b2e0a9273f\") " pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:28:56.915673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:56.915646    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4r4\""
Sep 16 17:28:56.923552 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:56.923536    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bfp6l"
Sep 16 17:28:57.193354 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:57.193277    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bfp6l"]
Sep 16 17:28:57.196982 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:28:57.196953    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4304ee43_101e_44b9_bfc7_87b2e0a9273f.slice/crio-51593c858cb73179e98dbdaf259566d8f12a1dcd24637b4b010f7879ec877e5e WatchSource:0}: Error finding container 51593c858cb73179e98dbdaf259566d8f12a1dcd24637b4b010f7879ec877e5e: Status 404 returned error can't find the container with id 51593c858cb73179e98dbdaf259566d8f12a1dcd24637b4b010f7879ec877e5e
Sep 16 17:28:57.990775 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:57.990738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bfp6l" event={"ID":"4304ee43-101e-44b9-bfc7-87b2e0a9273f","Type":"ContainerStarted","Data":"51593c858cb73179e98dbdaf259566d8f12a1dcd24637b4b010f7879ec877e5e"}
Sep 16 17:28:58.996449 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:58.996411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bfp6l" event={"ID":"4304ee43-101e-44b9-bfc7-87b2e0a9273f","Type":"ContainerStarted","Data":"4e102582084ce0d95120656f8d52d50a74cf58d6ac7b7b88759ff9e415e4a675"}
Sep 16 17:28:58.996861 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:58.996463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bfp6l" event={"ID":"4304ee43-101e-44b9-bfc7-87b2e0a9273f","Type":"ContainerStarted","Data":"0af459ef4d8d3578659def1c1b4ff55e82d3f862390b2c80228ab9d1f580050b"}
Sep 16 17:28:59.022537 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:28:59.022492    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bfp6l" podStartSLOduration=253.171245325 podStartE2EDuration="4m14.022476933s" totalImagesPullingTime="851.231608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:28:59.021758867 +0000 UTC m=+254.430876117" watchObservedRunningTime="2026-09-16 17:28:59.022476933 +0000 UTC m=+254.431594180"
Sep 16 17:29:08.339332 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.339263    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bf7467cb4-76tmx"]
Sep 16 17:29:08.339717 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.339552    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4913016-d411-4760-b16d-dde5946a416b" containerName="console"
Sep 16 17:29:08.339717 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.339564    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4913016-d411-4760-b16d-dde5946a416b" containerName="console"
Sep 16 17:29:08.339717 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.339625    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4913016-d411-4760-b16d-dde5946a416b" containerName="console"
Sep 16 17:29:08.343307 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.343291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.363042 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.363012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf7467cb4-76tmx"]
Sep 16 17:29:08.400387 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-console-config\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.400480 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-oauth-config\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.400480 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-oauth-serving-cert\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.400480 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w4q2\" (UniqueName: \"kubernetes.io/projected/f70d38d3-0da8-476e-8ef0-45991a96349e-kube-api-access-4w4q2\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.400602 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-serving-cert\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.400602 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-service-ca\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.400602 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.400528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-trusted-ca-bundle\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.501807 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-console-config\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.501908 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-oauth-config\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.501908 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-oauth-serving-cert\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.501908 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w4q2\" (UniqueName: \"kubernetes.io/projected/f70d38d3-0da8-476e-8ef0-45991a96349e-kube-api-access-4w4q2\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.501908 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-serving-cert\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.502090 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-service-ca\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.502090 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.501936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-trusted-ca-bundle\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.502560 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.502539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-oauth-serving-cert\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.502697 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.502679    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-service-ca\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.502748 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.502691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-console-config\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.502795 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.502780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-trusted-ca-bundle\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.504796 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.504771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-serving-cert\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.504796 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.504782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-oauth-config\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.512096 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.512076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w4q2\" (UniqueName: \"kubernetes.io/projected/f70d38d3-0da8-476e-8ef0-45991a96349e-kube-api-access-4w4q2\") pod \"console-5bf7467cb4-76tmx\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") " pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.652310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.652219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:08.929019 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:08.928953    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bf7467cb4-76tmx"]
Sep 16 17:29:08.932275 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:29:08.932251    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70d38d3_0da8_476e_8ef0_45991a96349e.slice/crio-241c11f93362bb184ca8c611e1126a559300d91338df029b9c2766e894c6226d WatchSource:0}: Error finding container 241c11f93362bb184ca8c611e1126a559300d91338df029b9c2766e894c6226d: Status 404 returned error can't find the container with id 241c11f93362bb184ca8c611e1126a559300d91338df029b9c2766e894c6226d
Sep 16 17:29:09.038043 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:09.038010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf7467cb4-76tmx" event={"ID":"f70d38d3-0da8-476e-8ef0-45991a96349e","Type":"ContainerStarted","Data":"b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863"}
Sep 16 17:29:09.038215 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:09.038055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf7467cb4-76tmx" event={"ID":"f70d38d3-0da8-476e-8ef0-45991a96349e","Type":"ContainerStarted","Data":"241c11f93362bb184ca8c611e1126a559300d91338df029b9c2766e894c6226d"}
Sep 16 17:29:09.062159 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:09.062111    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bf7467cb4-76tmx" podStartSLOduration=1.0620956480000001 podStartE2EDuration="1.062095648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:29:09.061458639 +0000 UTC m=+264.470575887" watchObservedRunningTime="2026-09-16 17:29:09.062095648 +0000 UTC m=+264.471212895"
Sep 16 17:29:18.652501 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:18.652452    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:18.652986 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:18.652545    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:18.657209 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:18.657187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:19.074847 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:19.074821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:29:19.127156 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:19.127127    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7695756959-bk6dn"]
Sep 16 17:29:44.145577 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.145515    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7695756959-bk6dn" podUID="16913a21-4551-4fcc-bcd7-afba75ad821c" containerName="console" containerID="cri-o://765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66" gracePeriod=15
Sep 16 17:29:44.381139 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.381113    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:29:44.457258 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457233    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-trusted-ca-bundle\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457398 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-oauth-serving-cert\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457398 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457318    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fnjk\" (UniqueName: \"kubernetes.io/projected/16913a21-4551-4fcc-bcd7-afba75ad821c-kube-api-access-2fnjk\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-service-ca\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457468    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-serving-cert\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457492    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-oauth-config\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-console-config\") pod \"16913a21-4551-4fcc-bcd7-afba75ad821c\" (UID: \"16913a21-4551-4fcc-bcd7-afba75ad821c\") "
Sep 16 17:29:44.457768 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:44.457768 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457753    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:44.457873 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.457848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-service-ca" (OuterVolumeSpecName: "service-ca") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:44.458035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.458013    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-console-config" (OuterVolumeSpecName: "console-config") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:29:44.459421 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.459395    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16913a21-4551-4fcc-bcd7-afba75ad821c-kube-api-access-2fnjk" (OuterVolumeSpecName: "kube-api-access-2fnjk") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "kube-api-access-2fnjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:29:44.459525 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.459498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:44.459525 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.459513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "16913a21-4551-4fcc-bcd7-afba75ad821c" (UID: "16913a21-4551-4fcc-bcd7-afba75ad821c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:29:44.559012 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.558978    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-oauth-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.559012 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.559010    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnjk\" (UniqueName: \"kubernetes.io/projected/16913a21-4551-4fcc-bcd7-afba75ad821c-kube-api-access-2fnjk\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.559012 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.559022    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-service-ca\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.559180 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.559031    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.559180 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.559039    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16913a21-4551-4fcc-bcd7-afba75ad821c-console-oauth-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.559180 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.559047    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-console-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.559180 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.559055    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16913a21-4551-4fcc-bcd7-afba75ad821c-trusted-ca-bundle\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:29:44.999440 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:44.999386    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 17:29:45.169640 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.169604    2560 generic.go:356] "Generic (PLEG): container finished" podID="16913a21-4551-4fcc-bcd7-afba75ad821c" containerID="765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66" exitCode=0
Sep 16 17:29:45.170035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.169653    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7695756959-bk6dn"
Sep 16 17:29:45.170035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.169685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695756959-bk6dn" event={"ID":"16913a21-4551-4fcc-bcd7-afba75ad821c","Type":"ContainerDied","Data":"765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66"}
Sep 16 17:29:45.170035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.169737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7695756959-bk6dn" event={"ID":"16913a21-4551-4fcc-bcd7-afba75ad821c","Type":"ContainerDied","Data":"089996496a384206eb3e3f289b1b33b29f7f305c719faba6f5e9dc53bd75ba07"}
Sep 16 17:29:45.170035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.169758    2560 scope.go:117] "RemoveContainer" containerID="765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66"
Sep 16 17:29:45.177486 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.177466    2560 scope.go:117] "RemoveContainer" containerID="765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66"
Sep 16 17:29:45.177768 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:29:45.177746    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66\": container with ID starting with 765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66 not found: ID does not exist" containerID="765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66"
Sep 16 17:29:45.177834 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.177775    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66"} err="failed to get container status \"765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66\": rpc error: code = NotFound desc = could not find container \"765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66\": container with ID starting with 765aaf9c75434adaf7b5020111c9dd737d97adca598058033c279b1a7d009e66 not found: ID does not exist"
Sep 16 17:29:45.190763 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.190742    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7695756959-bk6dn"]
Sep 16 17:29:45.200090 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:45.198242    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7695756959-bk6dn"]
Sep 16 17:29:47.116635 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:29:47.116579    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16913a21-4551-4fcc-bcd7-afba75ad821c" path="/var/lib/kubelet/pods/16913a21-4551-4fcc-bcd7-afba75ad821c/volumes"
Sep 16 17:30:15.584956 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.584926    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-774df85894-l9r66"]
Sep 16 17:30:15.585398 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.585197    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16913a21-4551-4fcc-bcd7-afba75ad821c" containerName="console"
Sep 16 17:30:15.585398 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.585209    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="16913a21-4551-4fcc-bcd7-afba75ad821c" containerName="console"
Sep 16 17:30:15.585398 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.585264    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="16913a21-4551-4fcc-bcd7-afba75ad821c" containerName="console"
Sep 16 17:30:15.587899 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.587882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.609057 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.609031    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774df85894-l9r66"]
Sep 16 17:30:15.665857 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.665827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsz8p\" (UniqueName: \"kubernetes.io/projected/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-kube-api-access-wsz8p\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.665857 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.665858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-trusted-ca-bundle\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.666016 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.665878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-config\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.666016 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.665895    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-service-ca\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.666016 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.665954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-oauth-config\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.666016 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.665981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-serving-cert\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.666153 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.666018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-oauth-serving-cert\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.766893 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.766862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsz8p\" (UniqueName: \"kubernetes.io/projected/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-kube-api-access-wsz8p\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.766893 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.766897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-trusted-ca-bundle\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.766918    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-config\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.766935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-service-ca\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.766973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-oauth-config\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.766991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-serving-cert\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.767020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-oauth-serving-cert\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767789 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.767761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-config\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767969 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.767774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-oauth-serving-cert\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.767969 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.767790    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-service-ca\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.768108 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.768072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-trusted-ca-bundle\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.769606 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.769570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-serving-cert\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.770012 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.769993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-oauth-config\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.775112 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.775080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsz8p\" (UniqueName: \"kubernetes.io/projected/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-kube-api-access-wsz8p\") pod \"console-774df85894-l9r66\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") " pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:15.896836 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:15.896764    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:16.169536 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:16.169377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-774df85894-l9r66"]
Sep 16 17:30:16.172076 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:30:16.172047    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba35a62_c09b_4ec5_bf5c_885f8e43aea7.slice/crio-6f24d42d791c65f984a6b8e2e9aab399332770b4a6e215ca805074a498fdbdde WatchSource:0}: Error finding container 6f24d42d791c65f984a6b8e2e9aab399332770b4a6e215ca805074a498fdbdde: Status 404 returned error can't find the container with id 6f24d42d791c65f984a6b8e2e9aab399332770b4a6e215ca805074a498fdbdde
Sep 16 17:30:16.280267 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:16.280234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774df85894-l9r66" event={"ID":"dba35a62-c09b-4ec5-bf5c-885f8e43aea7","Type":"ContainerStarted","Data":"2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97"}
Sep 16 17:30:16.280420 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:16.280288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774df85894-l9r66" event={"ID":"dba35a62-c09b-4ec5-bf5c-885f8e43aea7","Type":"ContainerStarted","Data":"6f24d42d791c65f984a6b8e2e9aab399332770b4a6e215ca805074a498fdbdde"}
Sep 16 17:30:16.299473 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:16.299429    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-774df85894-l9r66" podStartSLOduration=1.2994124280000001 podStartE2EDuration="1.299412428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:30:16.29771369 +0000 UTC m=+331.706830948" watchObservedRunningTime="2026-09-16 17:30:16.299412428 +0000 UTC m=+331.708529675"
Sep 16 17:30:25.897397 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:25.897358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:25.897778 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:25.897440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:25.901723 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:25.901702    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:26.325277 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:26.325249    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:30:26.372399 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:26.372371    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf7467cb4-76tmx"]
Sep 16 17:30:51.391573 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.391506    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bf7467cb4-76tmx" podUID="f70d38d3-0da8-476e-8ef0-45991a96349e" containerName="console" containerID="cri-o://b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863" gracePeriod=15
Sep 16 17:30:51.628650 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.628627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:30:51.725649 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725626    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-oauth-config\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.725786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725673    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w4q2\" (UniqueName: \"kubernetes.io/projected/f70d38d3-0da8-476e-8ef0-45991a96349e-kube-api-access-4w4q2\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.725786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725699    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-oauth-serving-cert\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.725786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725761    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-console-config\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.725786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-trusted-ca-bundle\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.725937 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-service-ca\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.725937 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.725866    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-serving-cert\") pod \"f70d38d3-0da8-476e-8ef0-45991a96349e\" (UID: \"f70d38d3-0da8-476e-8ef0-45991a96349e\") "
Sep 16 17:30:51.726153 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.726117    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-console-config" (OuterVolumeSpecName: "console-config") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:51.726153 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.726136    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:51.726322 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.726212    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:51.726322 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.726205    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-service-ca" (OuterVolumeSpecName: "service-ca") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:30:51.727747 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.727725    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:30:51.727962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.727941    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:30:51.728055 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.728040    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70d38d3-0da8-476e-8ef0-45991a96349e-kube-api-access-4w4q2" (OuterVolumeSpecName: "kube-api-access-4w4q2") pod "f70d38d3-0da8-476e-8ef0-45991a96349e" (UID: "f70d38d3-0da8-476e-8ef0-45991a96349e"). InnerVolumeSpecName "kube-api-access-4w4q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:30:51.826673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826642    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w4q2\" (UniqueName: \"kubernetes.io/projected/f70d38d3-0da8-476e-8ef0-45991a96349e-kube-api-access-4w4q2\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.826673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826669    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-oauth-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.826673 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826680    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-console-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.826918 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826690    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-trusted-ca-bundle\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.826918 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826698    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f70d38d3-0da8-476e-8ef0-45991a96349e-service-ca\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.826918 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826706    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:51.826918 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:51.826714    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f70d38d3-0da8-476e-8ef0-45991a96349e-console-oauth-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:30:52.414988 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.414951    2560 generic.go:356] "Generic (PLEG): container finished" podID="f70d38d3-0da8-476e-8ef0-45991a96349e" containerID="b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863" exitCode=0
Sep 16 17:30:52.415356 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.414999    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bf7467cb4-76tmx"
Sep 16 17:30:52.415356 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.415037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf7467cb4-76tmx" event={"ID":"f70d38d3-0da8-476e-8ef0-45991a96349e","Type":"ContainerDied","Data":"b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863"}
Sep 16 17:30:52.415356 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.415092    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bf7467cb4-76tmx" event={"ID":"f70d38d3-0da8-476e-8ef0-45991a96349e","Type":"ContainerDied","Data":"241c11f93362bb184ca8c611e1126a559300d91338df029b9c2766e894c6226d"}
Sep 16 17:30:52.415356 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.415114    2560 scope.go:117] "RemoveContainer" containerID="b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863"
Sep 16 17:30:52.423816 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.423797    2560 scope.go:117] "RemoveContainer" containerID="b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863"
Sep 16 17:30:52.424085 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:30:52.424066    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863\": container with ID starting with b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863 not found: ID does not exist" containerID="b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863"
Sep 16 17:30:52.424156 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.424097    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863"} err="failed to get container status \"b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863\": rpc error: code = NotFound desc = could not find container \"b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863\": container with ID starting with b18e8cf518348752512d43671a3a7dc46d1865fad050945d03be1826a5891863 not found: ID does not exist"
Sep 16 17:30:52.437130 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.437106    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bf7467cb4-76tmx"]
Sep 16 17:30:52.441807 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:52.441788    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bf7467cb4-76tmx"]
Sep 16 17:30:53.116675 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:30:53.116641    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f70d38d3-0da8-476e-8ef0-45991a96349e" path="/var/lib/kubelet/pods/f70d38d3-0da8-476e-8ef0-45991a96349e/volumes"
Sep 16 17:32:36.182229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.182155    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bd4bf99f8-clfv8"]
Sep 16 17:32:36.182688 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.182419    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f70d38d3-0da8-476e-8ef0-45991a96349e" containerName="console"
Sep 16 17:32:36.182688 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.182433    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f70d38d3-0da8-476e-8ef0-45991a96349e" containerName="console"
Sep 16 17:32:36.182688 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.182490    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f70d38d3-0da8-476e-8ef0-45991a96349e" containerName="console"
Sep 16 17:32:36.185156 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.185141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.210632 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.210607    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd4bf99f8-clfv8"]
Sep 16 17:32:36.250513 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-config\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.250627 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-service-ca\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.250627 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-oauth-config\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.250627 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-serving-cert\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.250729 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-trusted-ca-bundle\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.250729 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-oauth-serving-cert\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.250729 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.250718    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzmv\" (UniqueName: \"kubernetes.io/projected/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-kube-api-access-2kzmv\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.351844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-config\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.351844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-service-ca\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352033 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-oauth-config\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352033 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-serving-cert\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352033 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351928    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-trusted-ca-bundle\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352033 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351958    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-oauth-serving-cert\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352033 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.351997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzmv\" (UniqueName: \"kubernetes.io/projected/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-kube-api-access-2kzmv\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352812 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.352784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-config\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352938 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.352823    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-service-ca\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352938 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.352904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-trusted-ca-bundle\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.352938 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.352927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-oauth-serving-cert\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.354518 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.354496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-serving-cert\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.354618 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.354539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-console-oauth-config\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.359868 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.359821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzmv\" (UniqueName: \"kubernetes.io/projected/c9e82abe-34c1-4578-9f02-ef0fcaab51a7-kube-api-access-2kzmv\") pod \"console-5bd4bf99f8-clfv8\" (UID: \"c9e82abe-34c1-4578-9f02-ef0fcaab51a7\") " pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.494094 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.494068    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:36.768080 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:36.767994    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd4bf99f8-clfv8"]
Sep 16 17:32:36.771659 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:32:36.771629    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e82abe_34c1_4578_9f02_ef0fcaab51a7.slice/crio-d60451b8627ff1c656bcdecfd1598d9f4fcd1ac72b85cd949697572cf2442198 WatchSource:0}: Error finding container d60451b8627ff1c656bcdecfd1598d9f4fcd1ac72b85cd949697572cf2442198: Status 404 returned error can't find the container with id d60451b8627ff1c656bcdecfd1598d9f4fcd1ac72b85cd949697572cf2442198
Sep 16 17:32:37.780942 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:37.780902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd4bf99f8-clfv8" event={"ID":"c9e82abe-34c1-4578-9f02-ef0fcaab51a7","Type":"ContainerStarted","Data":"7690e2f2f9205f6ad32b16c2a21237e773ebccff0b336389ba301829b09ccfac"}
Sep 16 17:32:37.781309 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:37.780961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd4bf99f8-clfv8" event={"ID":"c9e82abe-34c1-4578-9f02-ef0fcaab51a7","Type":"ContainerStarted","Data":"d60451b8627ff1c656bcdecfd1598d9f4fcd1ac72b85cd949697572cf2442198"}
Sep 16 17:32:37.803573 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:37.803525    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bd4bf99f8-clfv8" podStartSLOduration=1.803510942 podStartE2EDuration="1.803510942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:32:37.803324038 +0000 UTC m=+473.212441287" watchObservedRunningTime="2026-09-16 17:32:37.803510942 +0000 UTC m=+473.212628183"
Sep 16 17:32:46.494780 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:46.494697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:46.494780 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:46.494756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:46.499350 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:46.499332    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:46.816152 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:46.816075    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bd4bf99f8-clfv8"
Sep 16 17:32:46.875792 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:32:46.875762    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774df85894-l9r66"]
Sep 16 17:33:11.894482 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:11.894446    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-774df85894-l9r66" podUID="dba35a62-c09b-4ec5-bf5c-885f8e43aea7" containerName="console" containerID="cri-o://2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97" gracePeriod=15
Sep 16 17:33:12.120831 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.120809    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:33:12.217444 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217420    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-serving-cert\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-oauth-serving-cert\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217470    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-config\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217490    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-oauth-config\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217514    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-trusted-ca-bundle\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217563    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-service-ca\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217609 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217604    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsz8p\" (UniqueName: \"kubernetes.io/projected/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-kube-api-access-wsz8p\") pod \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\" (UID: \"dba35a62-c09b-4ec5-bf5c-885f8e43aea7\") "
Sep 16 17:33:12.217950 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217921    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-config" (OuterVolumeSpecName: "console-config") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:12.218020 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.217991    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:12.218020 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.218000    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-service-ca" (OuterVolumeSpecName: "service-ca") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:12.218437 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.218409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:33:12.219671 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.219645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:33:12.219776 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.219757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:33:12.219886 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.219865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-kube-api-access-wsz8p" (OuterVolumeSpecName: "kube-api-access-wsz8p") pod "dba35a62-c09b-4ec5-bf5c-885f8e43aea7" (UID: "dba35a62-c09b-4ec5-bf5c-885f8e43aea7"). InnerVolumeSpecName "kube-api-access-wsz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:33:12.318552 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318526    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-trusted-ca-bundle\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.318552 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318548    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-service-ca\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.318552 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318557    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsz8p\" (UniqueName: \"kubernetes.io/projected/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-kube-api-access-wsz8p\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.318740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318566    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.318740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318575    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-oauth-serving-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.318740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318583    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.318740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.318611    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dba35a62-c09b-4ec5-bf5c-885f8e43aea7-console-oauth-config\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:33:12.907459 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.907425    2560 generic.go:356] "Generic (PLEG): container finished" podID="dba35a62-c09b-4ec5-bf5c-885f8e43aea7" containerID="2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97" exitCode=0
Sep 16 17:33:12.907876 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.907468    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-774df85894-l9r66"
Sep 16 17:33:12.907876 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.907510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774df85894-l9r66" event={"ID":"dba35a62-c09b-4ec5-bf5c-885f8e43aea7","Type":"ContainerDied","Data":"2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97"}
Sep 16 17:33:12.907876 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.907557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-774df85894-l9r66" event={"ID":"dba35a62-c09b-4ec5-bf5c-885f8e43aea7","Type":"ContainerDied","Data":"6f24d42d791c65f984a6b8e2e9aab399332770b4a6e215ca805074a498fdbdde"}
Sep 16 17:33:12.907876 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.907577    2560 scope.go:117] "RemoveContainer" containerID="2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97"
Sep 16 17:33:12.915800 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.915784    2560 scope.go:117] "RemoveContainer" containerID="2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97"
Sep 16 17:33:12.916031 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:33:12.916005    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97\": container with ID starting with 2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97 not found: ID does not exist" containerID="2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97"
Sep 16 17:33:12.916093 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.916043    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97"} err="failed to get container status \"2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97\": rpc error: code = NotFound desc = could not find container \"2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97\": container with ID starting with 2c70226c76fdd28fe6cd6347f4d14b31efa01d46e9f1a74f25eca26e0802ee97 not found: ID does not exist"
Sep 16 17:33:12.932853 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.932832    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-774df85894-l9r66"]
Sep 16 17:33:12.936481 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:12.936462    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-774df85894-l9r66"]
Sep 16 17:33:13.118114 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:13.118083    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dba35a62-c09b-4ec5-bf5c-885f8e43aea7" path="/var/lib/kubelet/pods/dba35a62-c09b-4ec5-bf5c-885f8e43aea7/volumes"
Sep 16 17:33:15.590515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.590482    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pz72s"]
Sep 16 17:33:15.590891 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.590749    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dba35a62-c09b-4ec5-bf5c-885f8e43aea7" containerName="console"
Sep 16 17:33:15.590891 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.590759    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dba35a62-c09b-4ec5-bf5c-885f8e43aea7" containerName="console"
Sep 16 17:33:15.590891 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.590814    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dba35a62-c09b-4ec5-bf5c-885f8e43aea7" containerName="console"
Sep 16 17:33:15.594705 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.594683    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:15.597260 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.597241    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-q4dsj\""
Sep 16 17:33:15.597820 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.597792    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 17:33:15.597820 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.597808    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 17:33:15.597989 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.597850    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 17:33:15.597989 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.597949    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 17:33:15.616730 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.616704    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pz72s"]
Sep 16 17:33:15.742805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.742777    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-certificates\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:15.742805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.742810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-kube-api-access-nsqwm\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:15.843696 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.843615    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-certificates\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:15.843839 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.843700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-kube-api-access-nsqwm\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:15.843839 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:33:15.843719    2560 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 16 17:33:15.843839 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:33:15.843739    2560 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-pz72s: secret "keda-admission-webhooks-certs" not found
Sep 16 17:33:15.843839 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:33:15.843792    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-certificates podName:7d300784-200e-4ad3-9cd8-51c2279639c5 nodeName:}" failed. No retries permitted until 2026-09-16 17:33:16.34377544 +0000 UTC m=+511.752892665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-certificates") pod "keda-admission-6c4d864b86-pz72s" (UID: "7d300784-200e-4ad3-9cd8-51c2279639c5") : secret "keda-admission-webhooks-certs" not found
Sep 16 17:33:15.854511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:15.854482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsqwm\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-kube-api-access-nsqwm\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:16.348761 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:16.348726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-certificates\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:16.351177 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:16.351149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7d300784-200e-4ad3-9cd8-51c2279639c5-certificates\") pod \"keda-admission-6c4d864b86-pz72s\" (UID: \"7d300784-200e-4ad3-9cd8-51c2279639c5\") " pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:16.504287 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:16.504258    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:16.793684 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:16.793656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-pz72s"]
Sep 16 17:33:16.797133 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:33:16.797097    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d300784_200e_4ad3_9cd8_51c2279639c5.slice/crio-10e4f723b2bad3af1e3d0f6afd4d682a1a0069766fea25daad95abf3b41d2f0d WatchSource:0}: Error finding container 10e4f723b2bad3af1e3d0f6afd4d682a1a0069766fea25daad95abf3b41d2f0d: Status 404 returned error can't find the container with id 10e4f723b2bad3af1e3d0f6afd4d682a1a0069766fea25daad95abf3b41d2f0d
Sep 16 17:33:16.798854 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:16.798837    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:33:16.923571 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:16.923522    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-pz72s" event={"ID":"7d300784-200e-4ad3-9cd8-51c2279639c5","Type":"ContainerStarted","Data":"10e4f723b2bad3af1e3d0f6afd4d682a1a0069766fea25daad95abf3b41d2f0d"}
Sep 16 17:33:19.942343 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:19.942306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-pz72s" event={"ID":"7d300784-200e-4ad3-9cd8-51c2279639c5","Type":"ContainerStarted","Data":"0fca056295c2984ae7c837a6c276d3adbd6fa8695fd22a676e7aab3e8245ee59"}
Sep 16 17:33:19.942740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:19.942409    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:33:19.963680 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:19.963627    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-pz72s" podStartSLOduration=2.326809214 podStartE2EDuration="4.963609873s" totalImagesPullingTime="2.636800659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:33:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:33:19.963286293 +0000 UTC m=+515.372403540" watchObservedRunningTime="2026-09-16 17:33:19.963609873 +0000 UTC m=+515.372727122"
Sep 16 17:33:40.948977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:33:40.948897    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-pz72s"
Sep 16 17:34:29.881191 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:29.881157    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"]
Sep 16 17:34:29.883356 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:29.883341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:29.886479 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:29.886449    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"cert-manager-cainjector-559cdd8c87-6v9cf\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" podUID="3af49011-8b09-40e3-b245-e63f995b801d" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:29.886753 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:29.886725    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cert-manager-cainjector-dockercfg-2qk9r\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2qk9r\"" type="*v1.Secret"
Sep 16 17:34:29.886835 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:29.886808    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 17:34:29.886835 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:29.886813    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"cert-manager\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 17:34:29.981126 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:29.981097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:29.981244 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:29.981154    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv72\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-kube-api-access-cfv72\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:30.031014 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:30.030981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"]
Sep 16 17:34:30.081541 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:30.081510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv72\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-kube-api-access-cfv72\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:30.081706 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:30.081561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:30.151624 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:30.151534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:31.153865 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:31.153831    2560 projected.go:291] Couldn't get configMap cert-manager/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:34:31.223762 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:31.223728    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 17:34:31.223967 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:31.223951    2560 projected.go:196] Error preparing data for projected volume kube-api-access-cfv72 for pod cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf: failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:34:31.224058 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:34:31.224048    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-kube-api-access-cfv72 podName:3af49011-8b09-40e3-b245-e63f995b801d nodeName:}" failed. No retries permitted until 2026-09-16 17:34:31.724019528 +0000 UTC m=+587.133136755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cfv72" (UniqueName: "kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-kube-api-access-cfv72") pod "cert-manager-cainjector-559cdd8c87-6v9cf" (UID: "3af49011-8b09-40e3-b245-e63f995b801d") : failed to sync configmap cache: timed out waiting for the condition
Sep 16 17:34:31.227459 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:31.227437    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 17:34:31.421436 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:31.421351    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2qk9r\""
Sep 16 17:34:31.795177 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:31.795129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv72\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-kube-api-access-cfv72\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:31.797655 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:31.797623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv72\" (UniqueName: \"kubernetes.io/projected/3af49011-8b09-40e3-b245-e63f995b801d-kube-api-access-cfv72\") pod \"cert-manager-cainjector-559cdd8c87-6v9cf\" (UID: \"3af49011-8b09-40e3-b245-e63f995b801d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:31.991997 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:31.991965    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"
Sep 16 17:34:32.289270 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:32.289235    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf"]
Sep 16 17:34:32.292690 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:34:32.292662    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af49011_8b09_40e3_b245_e63f995b801d.slice/crio-838a18f29585adfc6e6ffebea17ab063f55315d47c01a9330f1567ac5dd125c1 WatchSource:0}: Error finding container 838a18f29585adfc6e6ffebea17ab063f55315d47c01a9330f1567ac5dd125c1: Status 404 returned error can't find the container with id 838a18f29585adfc6e6ffebea17ab063f55315d47c01a9330f1567ac5dd125c1
Sep 16 17:34:33.203364 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:33.203325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf" event={"ID":"3af49011-8b09-40e3-b245-e63f995b801d","Type":"ContainerStarted","Data":"838a18f29585adfc6e6ffebea17ab063f55315d47c01a9330f1567ac5dd125c1"}
Sep 16 17:34:36.216606 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:36.216564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf" event={"ID":"3af49011-8b09-40e3-b245-e63f995b801d","Type":"ContainerStarted","Data":"13ee26fb536237c5d3a8a60eb920a2fd8075b2f6799314b3645f89fbdef94c3e"}
Sep 16 17:34:47.764730 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.764681    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-6v9cf" podStartSLOduration=15.104566472 podStartE2EDuration="18.764665519s" totalImagesPullingTime="3.660099047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:36.238987663 +0000 UTC m=+591.648104917" watchObservedRunningTime="2026-09-16 17:34:47.764665519 +0000 UTC m=+603.173782768"
Sep 16 17:34:47.765551 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.765528    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f5jnt"]
Sep 16 17:34:47.767434 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.767419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:47.769323 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.769303    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-t6g69\""
Sep 16 17:34:47.781828 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.781806    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f5jnt"]
Sep 16 17:34:47.915527 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.915497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpjn\" (UniqueName: \"kubernetes.io/projected/6648a8f1-b9be-4f36-8432-bbba95259e02-kube-api-access-9lpjn\") pod \"cert-manager-55b4d954d5-f5jnt\" (UID: \"6648a8f1-b9be-4f36-8432-bbba95259e02\") " pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:47.915716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:47.915561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6648a8f1-b9be-4f36-8432-bbba95259e02-bound-sa-token\") pod \"cert-manager-55b4d954d5-f5jnt\" (UID: \"6648a8f1-b9be-4f36-8432-bbba95259e02\") " pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:48.016171 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:48.016085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6648a8f1-b9be-4f36-8432-bbba95259e02-bound-sa-token\") pod \"cert-manager-55b4d954d5-f5jnt\" (UID: \"6648a8f1-b9be-4f36-8432-bbba95259e02\") " pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:48.016171 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:48.016135    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpjn\" (UniqueName: \"kubernetes.io/projected/6648a8f1-b9be-4f36-8432-bbba95259e02-kube-api-access-9lpjn\") pod \"cert-manager-55b4d954d5-f5jnt\" (UID: \"6648a8f1-b9be-4f36-8432-bbba95259e02\") " pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:48.025007 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:48.024984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6648a8f1-b9be-4f36-8432-bbba95259e02-bound-sa-token\") pod \"cert-manager-55b4d954d5-f5jnt\" (UID: \"6648a8f1-b9be-4f36-8432-bbba95259e02\") " pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:48.025158 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:48.025140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpjn\" (UniqueName: \"kubernetes.io/projected/6648a8f1-b9be-4f36-8432-bbba95259e02-kube-api-access-9lpjn\") pod \"cert-manager-55b4d954d5-f5jnt\" (UID: \"6648a8f1-b9be-4f36-8432-bbba95259e02\") " pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:48.080177 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:48.080155    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-f5jnt"
Sep 16 17:34:48.349559 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:48.349479    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-f5jnt"]
Sep 16 17:34:48.352728 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:34:48.352695    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6648a8f1_b9be_4f36_8432_bbba95259e02.slice/crio-015805f3b27c5d8c2e3b6c080cbb4a04dc19d88e213b9b87032e36932d53a66d WatchSource:0}: Error finding container 015805f3b27c5d8c2e3b6c080cbb4a04dc19d88e213b9b87032e36932d53a66d: Status 404 returned error can't find the container with id 015805f3b27c5d8c2e3b6c080cbb4a04dc19d88e213b9b87032e36932d53a66d
Sep 16 17:34:49.263878 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:49.263841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f5jnt" event={"ID":"6648a8f1-b9be-4f36-8432-bbba95259e02","Type":"ContainerStarted","Data":"ed949e7986f489cc0d00bf8ba0b9366746dbf7ea661d41852d356679e98fd609"}
Sep 16 17:34:49.264243 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:49.263892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-f5jnt" event={"ID":"6648a8f1-b9be-4f36-8432-bbba95259e02","Type":"ContainerStarted","Data":"015805f3b27c5d8c2e3b6c080cbb4a04dc19d88e213b9b87032e36932d53a66d"}
Sep 16 17:34:49.285312 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:34:49.285264    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-f5jnt" podStartSLOduration=2.285248926 podStartE2EDuration="2.285248926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:34:49.284941154 +0000 UTC m=+604.694058401" watchObservedRunningTime="2026-09-16 17:34:49.285248926 +0000 UTC m=+604.694366175"
Sep 16 17:35:06.166922 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.166891    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"]
Sep 16 17:35:06.170314 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.170298    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.176925 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.176903    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-xxzcg\""
Sep 16 17:35:06.176925 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.176922    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 17:35:06.177068 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.176921    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 17:35:06.177399 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.177383    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 17:35:06.190677 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.190658    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"]
Sep 16 17:35:06.252856 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.252829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.252957 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.252859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5bd874-c917-4ebf-994b-7a0954556b05-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.252957 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.252882    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.252957 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.252903    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.252957 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.252939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.253117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.252977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.253117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.253014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.253117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.253057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.253117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.253092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m86n8\" (UniqueName: \"kubernetes.io/projected/1c5bd874-c917-4ebf-994b-7a0954556b05-kube-api-access-m86n8\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.353978 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.353946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354121 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.353986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m86n8\" (UniqueName: \"kubernetes.io/projected/1c5bd874-c917-4ebf-994b-7a0954556b05-kube-api-access-m86n8\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354121 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354232 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5bd874-c917-4ebf-994b-7a0954556b05-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354232 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354232 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354184    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354232 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354419 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354419 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354419 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354325    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-workload-certs\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354572 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354469    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-data\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354648 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354627    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-workload-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354713 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-credential-socket\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.354859 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.354836    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c5bd874-c917-4ebf-994b-7a0954556b05-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.356510 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.356491    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-envoy\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.356676 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.356660    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.364880 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.364858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c5bd874-c917-4ebf-994b-7a0954556b05-istio-token\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.367087 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.367068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m86n8\" (UniqueName: \"kubernetes.io/projected/1c5bd874-c917-4ebf-994b-7a0954556b05-kube-api-access-m86n8\") pod \"openshift-ai-inference-openshift-default-8699b59bbc-nl7lw\" (UID: \"1c5bd874-c917-4ebf-994b-7a0954556b05\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.479735 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.479713    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:06.760836 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:06.760762    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"]
Sep 16 17:35:06.763889 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:35:06.763861    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5bd874_c917_4ebf_994b_7a0954556b05.slice/crio-49a41d4b3116c024496c816f5c994e2683acf6479a7028412fe37d4c305b0d31 WatchSource:0}: Error finding container 49a41d4b3116c024496c816f5c994e2683acf6479a7028412fe37d4c305b0d31: Status 404 returned error can't find the container with id 49a41d4b3116c024496c816f5c994e2683acf6479a7028412fe37d4c305b0d31
Sep 16 17:35:07.330749 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:07.330672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" event={"ID":"1c5bd874-c917-4ebf-994b-7a0954556b05","Type":"ContainerStarted","Data":"49a41d4b3116c024496c816f5c994e2683acf6479a7028412fe37d4c305b0d31"}
Sep 16 17:35:09.761343 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:09.761301    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:35:09.761634 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:09.761389    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:35:09.761634 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:09.761416    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:35:10.346682 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:10.346645    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" event={"ID":"1c5bd874-c917-4ebf-994b-7a0954556b05","Type":"ContainerStarted","Data":"34d57eac54ad30fb5c7ef99d3fc3fd03c75ec1dad251a0edbd57d003702fb73a"}
Sep 16 17:35:10.370490 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:10.370440    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" podStartSLOduration=1.375232642 podStartE2EDuration="4.370423498s" totalImagesPullingTime="2.995190856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:10.368963203 +0000 UTC m=+625.778080451" watchObservedRunningTime="2026-09-16 17:35:10.370423498 +0000 UTC m=+625.779540746"
Sep 16 17:35:10.480256 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:10.480222    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:10.489754 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:10.489723    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:11.350861 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:11.350827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:11.351860 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:11.351842    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:35:12.559105 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.559068    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"]
Sep 16 17:35:12.562362 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.562341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.565911 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.565884    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 17:35:12.566321 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.566302    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 17:35:12.566549 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.566536    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-whtnm\""
Sep 16 17:35:12.566794 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.566765    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 17:35:12.567130 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.567115    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 17:35:12.567624 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.567605    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 17:35:12.606313 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.606282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-metrics-cert\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.606313 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.606314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-cert\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.606557 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.606341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds2nb\" (UniqueName: \"kubernetes.io/projected/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-kube-api-access-ds2nb\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.606557 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.606417    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-manager-config\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.607932 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.607905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"]
Sep 16 17:35:12.707674 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.707635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds2nb\" (UniqueName: \"kubernetes.io/projected/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-kube-api-access-ds2nb\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.707884 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.707689    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-manager-config\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.707884 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.707735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-metrics-cert\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.707884 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.707753    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-cert\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.708433 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.708410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-manager-config\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.710458 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.710427    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-cert\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.710555 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.710435    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-metrics-cert\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.717222 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.717202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds2nb\" (UniqueName: \"kubernetes.io/projected/80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd-kube-api-access-ds2nb\") pod \"lws-controller-manager-86b645b5b5-hpbfr\" (UID: \"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd\") " pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:12.871618 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:12.871503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:13.153720 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:13.153634    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"]
Sep 16 17:35:13.156487 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:35:13.156460    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d5ca89_d28e_49dd_b5c7_8acbaab9b5cd.slice/crio-683a7c72e54902bcc8c2a16e4023779f0b7e17eccb427ed14df42a04ab7dd989 WatchSource:0}: Error finding container 683a7c72e54902bcc8c2a16e4023779f0b7e17eccb427ed14df42a04ab7dd989: Status 404 returned error can't find the container with id 683a7c72e54902bcc8c2a16e4023779f0b7e17eccb427ed14df42a04ab7dd989
Sep 16 17:35:13.360487 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:13.360453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr" event={"ID":"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd","Type":"ContainerStarted","Data":"683a7c72e54902bcc8c2a16e4023779f0b7e17eccb427ed14df42a04ab7dd989"}
Sep 16 17:35:16.379539 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:16.379504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr" event={"ID":"80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd","Type":"ContainerStarted","Data":"01417964f7c29bec2fa381cd7b670d66c884ef4c5c0c8927377a8a36243ffd88"}
Sep 16 17:35:16.379960 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:16.379580    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:16.399066 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:16.399018    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr" podStartSLOduration=1.379818544 podStartE2EDuration="4.399003812s" totalImagesPullingTime="3.019185268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:16.39848612 +0000 UTC m=+631.807603369" watchObservedRunningTime="2026-09-16 17:35:16.399003812 +0000 UTC m=+631.808121060"
Sep 16 17:35:27.385824 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:27.385790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86b645b5b5-hpbfr"
Sep 16 17:35:50.256633 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.256578    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"]
Sep 16 17:35:50.266789 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.266763    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.269336 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.269312    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 17:35:50.269469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.269338    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 17:35:50.269469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.269361    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 17:35:50.269582 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.269485    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 17:35:50.269582 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.269508    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2xswh\""
Sep 16 17:35:50.270085 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.270064    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"]
Sep 16 17:35:50.401463 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.401429    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.401739 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.401474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh457\" (UniqueName: \"kubernetes.io/projected/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-kube-api-access-vh457\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.401739 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.401574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.501968 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.501935    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.502132 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.501997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh457\" (UniqueName: \"kubernetes.io/projected/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-kube-api-access-vh457\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.502132 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.502044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.502726 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.502699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.504637 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.504610    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.510417 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.510360    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh457\" (UniqueName: \"kubernetes.io/projected/30f0160f-0edd-4747-bcf8-13d0a18a6ec2-kube-api-access-vh457\") pod \"kuadrant-console-plugin-fbdd4f445-5d8zg\" (UID: \"30f0160f-0edd-4747-bcf8-13d0a18a6ec2\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.576350 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.576328    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"
Sep 16 17:35:50.855608 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:50.855514    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg"]
Sep 16 17:35:50.859491 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:35:50.859465    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30f0160f_0edd_4747_bcf8_13d0a18a6ec2.slice/crio-8b958069d46c0619fef290ca48f61d3516b1076c0452b2aca987530a3c8f7fc2 WatchSource:0}: Error finding container 8b958069d46c0619fef290ca48f61d3516b1076c0452b2aca987530a3c8f7fc2: Status 404 returned error can't find the container with id 8b958069d46c0619fef290ca48f61d3516b1076c0452b2aca987530a3c8f7fc2
Sep 16 17:35:51.510282 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:51.510252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg" event={"ID":"30f0160f-0edd-4747-bcf8-13d0a18a6ec2","Type":"ContainerStarted","Data":"8b958069d46c0619fef290ca48f61d3516b1076c0452b2aca987530a3c8f7fc2"}
Sep 16 17:35:55.528407 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:55.528371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg" event={"ID":"30f0160f-0edd-4747-bcf8-13d0a18a6ec2","Type":"ContainerStarted","Data":"92d40e908d8efa652f84eb60845ecfdc04e65d554f13b37c00ef11d5aeba1092"}
Sep 16 17:35:55.544626 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:35:55.544566    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5d8zg" podStartSLOduration=1.029430558 podStartE2EDuration="5.544552272s" totalImagesPullingTime="4.515121714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:35:55.5437653 +0000 UTC m=+670.952882549" watchObservedRunningTime="2026-09-16 17:35:55.544552272 +0000 UTC m=+670.953669520"
Sep 16 17:36:35.425462 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.425428    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qg48b"]
Sep 16 17:36:35.445157 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.445122    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qg48b"]
Sep 16 17:36:35.445302 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.445209    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:35.447251 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.447229    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rbkn4\""
Sep 16 17:36:35.562860 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.562829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgt8k\" (UniqueName: \"kubernetes.io/projected/edb96c2e-71ca-49a8-9083-d02e391bfb10-kube-api-access-fgt8k\") pod \"authorino-7bcfd8f9cd-qg48b\" (UID: \"edb96c2e-71ca-49a8-9083-d02e391bfb10\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:35.663972 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.663943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgt8k\" (UniqueName: \"kubernetes.io/projected/edb96c2e-71ca-49a8-9083-d02e391bfb10-kube-api-access-fgt8k\") pod \"authorino-7bcfd8f9cd-qg48b\" (UID: \"edb96c2e-71ca-49a8-9083-d02e391bfb10\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:35.676896 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.676829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgt8k\" (UniqueName: \"kubernetes.io/projected/edb96c2e-71ca-49a8-9083-d02e391bfb10-kube-api-access-fgt8k\") pod \"authorino-7bcfd8f9cd-qg48b\" (UID: \"edb96c2e-71ca-49a8-9083-d02e391bfb10\") " pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:35.753920 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:35.753889    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:36.023028 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:36.023000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qg48b"]
Sep 16 17:36:36.026023 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:36:36.025994    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedb96c2e_71ca_49a8_9083_d02e391bfb10.slice/crio-ff8aa23145dcb36bbdebdcafbfd2bbef43af5361d08e9ab22e7a7d1cacfd8995 WatchSource:0}: Error finding container ff8aa23145dcb36bbdebdcafbfd2bbef43af5361d08e9ab22e7a7d1cacfd8995: Status 404 returned error can't find the container with id ff8aa23145dcb36bbdebdcafbfd2bbef43af5361d08e9ab22e7a7d1cacfd8995
Sep 16 17:36:36.678047 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:36.678012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b" event={"ID":"edb96c2e-71ca-49a8-9083-d02e391bfb10","Type":"ContainerStarted","Data":"ff8aa23145dcb36bbdebdcafbfd2bbef43af5361d08e9ab22e7a7d1cacfd8995"}
Sep 16 17:36:38.687469 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:38.687436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b" event={"ID":"edb96c2e-71ca-49a8-9083-d02e391bfb10","Type":"ContainerStarted","Data":"61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a"}
Sep 16 17:36:38.704653 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:38.704604    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b" podStartSLOduration=1.247911985 podStartE2EDuration="3.70457296s" totalImagesPullingTime="2.456660975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:38.703134199 +0000 UTC m=+714.112251447" watchObservedRunningTime="2026-09-16 17:36:38.70457296 +0000 UTC m=+714.113690208"
Sep 16 17:36:39.368630 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:39.368582    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qg48b"]
Sep 16 17:36:40.695425 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:40.695386    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b" podUID="edb96c2e-71ca-49a8-9083-d02e391bfb10" containerName="authorino" containerID="cri-o://61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a" gracePeriod=30
Sep 16 17:36:40.935208 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:40.935182    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:41.012656 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.012626    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgt8k\" (UniqueName: \"kubernetes.io/projected/edb96c2e-71ca-49a8-9083-d02e391bfb10-kube-api-access-fgt8k\") pod \"edb96c2e-71ca-49a8-9083-d02e391bfb10\" (UID: \"edb96c2e-71ca-49a8-9083-d02e391bfb10\") "
Sep 16 17:36:41.014743 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.014712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edb96c2e-71ca-49a8-9083-d02e391bfb10-kube-api-access-fgt8k" (OuterVolumeSpecName: "kube-api-access-fgt8k") pod "edb96c2e-71ca-49a8-9083-d02e391bfb10" (UID: "edb96c2e-71ca-49a8-9083-d02e391bfb10"). InnerVolumeSpecName "kube-api-access-fgt8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:36:41.113230 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.113190    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgt8k\" (UniqueName: \"kubernetes.io/projected/edb96c2e-71ca-49a8-9083-d02e391bfb10-kube-api-access-fgt8k\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:36:41.699970 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.699938    2560 generic.go:356] "Generic (PLEG): container finished" podID="edb96c2e-71ca-49a8-9083-d02e391bfb10" containerID="61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a" exitCode=0
Sep 16 17:36:41.700441 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.699980    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b"
Sep 16 17:36:41.700441 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.700023    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b" event={"ID":"edb96c2e-71ca-49a8-9083-d02e391bfb10","Type":"ContainerDied","Data":"61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a"}
Sep 16 17:36:41.700441 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.700076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-qg48b" event={"ID":"edb96c2e-71ca-49a8-9083-d02e391bfb10","Type":"ContainerDied","Data":"ff8aa23145dcb36bbdebdcafbfd2bbef43af5361d08e9ab22e7a7d1cacfd8995"}
Sep 16 17:36:41.700441 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.700098    2560 scope.go:117] "RemoveContainer" containerID="61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a"
Sep 16 17:36:41.707944 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.707923    2560 scope.go:117] "RemoveContainer" containerID="61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a"
Sep 16 17:36:41.708276 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:36:41.708254    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a\": container with ID starting with 61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a not found: ID does not exist" containerID="61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a"
Sep 16 17:36:41.708375 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.708281    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a"} err="failed to get container status \"61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a\": rpc error: code = NotFound desc = could not find container \"61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a\": container with ID starting with 61e028d8204a085a29dffa7b4648f15a31885ece6d861cbe8a81fab99718dc1a not found: ID does not exist"
Sep 16 17:36:41.718649 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.718626    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qg48b"]
Sep 16 17:36:41.719796 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:41.719777    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-qg48b"]
Sep 16 17:36:43.116847 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:43.116815    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edb96c2e-71ca-49a8-9083-d02e391bfb10" path="/var/lib/kubelet/pods/edb96c2e-71ca-49a8-9083-d02e391bfb10/volumes"
Sep 16 17:36:57.356092 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.356056    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5zj2x"]
Sep 16 17:36:57.356458 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.356347    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edb96c2e-71ca-49a8-9083-d02e391bfb10" containerName="authorino"
Sep 16 17:36:57.356458 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.356358    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="edb96c2e-71ca-49a8-9083-d02e391bfb10" containerName="authorino"
Sep 16 17:36:57.356458 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.356414    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="edb96c2e-71ca-49a8-9083-d02e391bfb10" containerName="authorino"
Sep 16 17:36:57.360485 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.360467    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.363730 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.363710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rbkn4\""
Sep 16 17:36:57.364219 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.364199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 17:36:57.371127 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.371103    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5zj2x"]
Sep 16 17:36:57.437745 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.437717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9ecf4ad-bb99-41dd-bac9-c90a542ddf94-tls-cert\") pod \"authorino-6d9898bd99-5zj2x\" (UID: \"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94\") " pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.437876 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.437768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4s7\" (UniqueName: \"kubernetes.io/projected/d9ecf4ad-bb99-41dd-bac9-c90a542ddf94-kube-api-access-ss4s7\") pod \"authorino-6d9898bd99-5zj2x\" (UID: \"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94\") " pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.538613 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.538565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9ecf4ad-bb99-41dd-bac9-c90a542ddf94-tls-cert\") pod \"authorino-6d9898bd99-5zj2x\" (UID: \"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94\") " pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.538756 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.538660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4s7\" (UniqueName: \"kubernetes.io/projected/d9ecf4ad-bb99-41dd-bac9-c90a542ddf94-kube-api-access-ss4s7\") pod \"authorino-6d9898bd99-5zj2x\" (UID: \"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94\") " pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.540898 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.540870    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d9ecf4ad-bb99-41dd-bac9-c90a542ddf94-tls-cert\") pod \"authorino-6d9898bd99-5zj2x\" (UID: \"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94\") " pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.547066 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.547043    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4s7\" (UniqueName: \"kubernetes.io/projected/d9ecf4ad-bb99-41dd-bac9-c90a542ddf94-kube-api-access-ss4s7\") pod \"authorino-6d9898bd99-5zj2x\" (UID: \"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94\") " pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.669630 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.669550    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d9898bd99-5zj2x"
Sep 16 17:36:57.938579 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:57.938508    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d9898bd99-5zj2x"]
Sep 16 17:36:57.941639 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:36:57.941611    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ecf4ad_bb99_41dd_bac9_c90a542ddf94.slice/crio-f7e47ffeb9aabca9c3dfd0592ccb5352c51b358f4226cbebc3229e266e78822a WatchSource:0}: Error finding container f7e47ffeb9aabca9c3dfd0592ccb5352c51b358f4226cbebc3229e266e78822a: Status 404 returned error can't find the container with id f7e47ffeb9aabca9c3dfd0592ccb5352c51b358f4226cbebc3229e266e78822a
Sep 16 17:36:58.767610 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:58.767563    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5zj2x" event={"ID":"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94","Type":"ContainerStarted","Data":"8e0d5bdcb29f65c9dc4233e6f99abe6768009bf87ace25a015d43503c0841c19"}
Sep 16 17:36:58.767977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:36:58.767620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d9898bd99-5zj2x" event={"ID":"d9ecf4ad-bb99-41dd-bac9-c90a542ddf94","Type":"ContainerStarted","Data":"f7e47ffeb9aabca9c3dfd0592ccb5352c51b358f4226cbebc3229e266e78822a"}
Sep 16 17:41:02.702003 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.701966    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c5bd874-c917-4ebf-994b-7a0954556b05" containerID="34d57eac54ad30fb5c7ef99d3fc3fd03c75ec1dad251a0edbd57d003702fb73a" exitCode=0
Sep 16 17:41:02.702508 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.702014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" event={"ID":"1c5bd874-c917-4ebf-994b-7a0954556b05","Type":"ContainerDied","Data":"34d57eac54ad30fb5c7ef99d3fc3fd03c75ec1dad251a0edbd57d003702fb73a"}
Sep 16 17:41:02.702508 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.702499    2560 scope.go:117] "RemoveContainer" containerID="34d57eac54ad30fb5c7ef99d3fc3fd03c75ec1dad251a0edbd57d003702fb73a"
Sep 16 17:41:02.703461 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.703428    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:41:02.703543 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.703500    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:41:02.703543 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.703536    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:41:02.727518 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:02.727466    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6d9898bd99-5zj2x" podStartSLOduration=245.055882103 podStartE2EDuration="4m5.72745273s" totalImagesPullingTime="671.570627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:36:58.805482511 +0000 UTC m=+734.214599760" watchObservedRunningTime="2026-09-16 17:41:02.72745273 +0000 UTC m=+978.136569989"
Sep 16 17:41:03.668196 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.668166    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"]
Sep 16 17:41:03.670432 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.670413    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.672556 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.672529    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"kube-root-ca.crt\""
Sep 16 17:41:03.672556 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.672528    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"default-dockercfg-smn6z\""
Sep 16 17:41:03.672763 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.672532    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:41:03.672763 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.672583    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"seaweedfs-s3-creds\""
Sep 16 17:41:03.672763 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.672612    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Sep 16 17:41:03.673011 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.672994    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-fac1c4ae\"/\"openshift-service-ca.crt\""
Sep 16 17:41:03.686335 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.686315    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"]
Sep 16 17:41:03.708073 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.708039    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" event={"ID":"1c5bd874-c917-4ebf-994b-7a0954556b05","Type":"ContainerStarted","Data":"fa4f5d9a541e72f742d2dd7d42474e5e032a34530f88f913a3554063d1b580c1"}
Sep 16 17:41:03.740570 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f79415e-a7ea-4c32-a5b8-96c351a0e591-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740570 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740802 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c57z\" (UniqueName: \"kubernetes.io/projected/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kube-api-access-5c57z\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740802 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740802 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740902 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740902 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.740902 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.740894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841600 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841600 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841807 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841636    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f79415e-a7ea-4c32-a5b8-96c351a0e591-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841807 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841663    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841807 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c57z\" (UniqueName: \"kubernetes.io/projected/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kube-api-access-5c57z\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841807 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841733    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841940 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.841940 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.841862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.842313 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.842286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.842351 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.842316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.842423 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.842399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.842538 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.842521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.843510 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.843481    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f79415e-a7ea-4c32-a5b8-96c351a0e591-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.844358 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.844336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.844632 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.844616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.851383 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.851365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c57z\" (UniqueName: \"kubernetes.io/projected/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kube-api-access-5c57z\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") " pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:03.980885 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:03.980853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:04.287772 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:04.287659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"]
Sep 16 17:41:04.290266 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:41:04.290229    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f79415e_a7ea_4c32_a5b8_96c351a0e591.slice/crio-8c073e02439e4ee6c430a58d19872001a3ccadd355190825445680dfa364c8d3 WatchSource:0}: Error finding container 8c073e02439e4ee6c430a58d19872001a3ccadd355190825445680dfa364c8d3: Status 404 returned error can't find the container with id 8c073e02439e4ee6c430a58d19872001a3ccadd355190825445680dfa364c8d3
Sep 16 17:41:04.292415 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:04.292389    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:41:04.480342 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:04.480296    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:41:04.480527 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:04.480503    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 16 17:41:04.480629 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:04.480575    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" podUID="1c5bd874-c917-4ebf-994b-7a0954556b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 16 17:41:04.713759 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:04.713718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" event={"ID":"3f79415e-a7ea-4c32-a5b8-96c351a0e591","Type":"ContainerStarted","Data":"8c073e02439e4ee6c430a58d19872001a3ccadd355190825445680dfa364c8d3"}
Sep 16 17:41:05.480450 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:05.480416    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 16 17:41:05.480632 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:05.480512    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" podUID="1c5bd874-c917-4ebf-994b-7a0954556b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 16 17:41:06.481040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:06.480962    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 16 17:41:06.481040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:06.481025    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" podUID="1c5bd874-c917-4ebf-994b-7a0954556b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 16 17:41:07.480517 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:07.480432    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 16 17:41:07.480677 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:07.480503    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" podUID="1c5bd874-c917-4ebf-994b-7a0954556b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 16 17:41:08.480472 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:08.480435    2560 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused" start-of-body=
Sep 16 17:41:08.480853 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:08.480490    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw" podUID="1c5bd874-c917-4ebf-994b-7a0954556b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.22:15021/healthz/ready\": dial tcp 10.134.0.22:15021: connect: connection refused"
Sep 16 17:41:08.754538 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:08.754407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" event={"ID":"3f79415e-a7ea-4c32-a5b8-96c351a0e591","Type":"ContainerStarted","Data":"ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0"}
Sep 16 17:41:09.484422 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:09.484381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:41:09.484925 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:09.484904    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:41:09.485557 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:09.485534    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-8699b59bbc-nl7lw"
Sep 16 17:41:11.767348 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:11.767317    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerID="ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0" exitCode=0
Sep 16 17:41:11.767874 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:11.767367    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" event={"ID":"3f79415e-a7ea-4c32-a5b8-96c351a0e591","Type":"ContainerDied","Data":"ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0"}
Sep 16 17:41:39.909800 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:39.909706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" event={"ID":"3f79415e-a7ea-4c32-a5b8-96c351a0e591","Type":"ContainerStarted","Data":"1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39"}
Sep 16 17:41:39.934940 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:39.934881    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podStartSLOduration=2.6913715529999997 podStartE2EDuration="36.934859882s" totalImagesPullingTime="32.02198235s" totalInitContainerRuntime="2.221505979s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:41:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:41:39.934047248 +0000 UTC m=+1015.343164496" watchObservedRunningTime="2026-09-16 17:41:39.934859882 +0000 UTC m=+1015.343977131"
Sep 16 17:41:39.981431 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:39.981392    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:43.981856 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:43.981822    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:41:43.983502 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:43.983474    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:41:53.981417 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:41:53.981381    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:42:03.981986 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:42:03.981938    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:42:13.981972 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:42:13.981930    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:42:23.981414 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:42:23.981371    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:42:33.981399 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:42:33.981350    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:42:43.981872 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:42:43.981775    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:42:53.981359 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:42:53.981307    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:43:03.981542 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:03.981487    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:43:13.981486 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:13.981438    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:43:23.982198 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:23.982153    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": dial tcp 10.134.0.27:8000: connect: connection refused"
Sep 16 17:43:33.991337 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:33.991305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:43:33.999077 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:33.999049    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:43:37.907493 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:37.907460    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"]
Sep 16 17:43:52.928767 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:52.928705    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main" containerID="cri-o://1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39" gracePeriod=45
Sep 16 17:43:53.668961 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.668937    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:43:53.779301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779269    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5c57z\" (UniqueName: \"kubernetes.io/projected/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kube-api-access-5c57z\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779302    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-model-cache\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779355    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-dshm\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779382    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f79415e-a7ea-4c32-a5b8-96c351a0e591-cabundle-cert\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779397    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tmp-dir\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779442    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kserve-provision-location\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779484    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tls-certs\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779519 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-home\") pod \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\" (UID: \"3f79415e-a7ea-4c32-a5b8-96c351a0e591\") "
Sep 16 17:43:53.779855 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779713    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:53.779912 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.779847    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f79415e-a7ea-4c32-a5b8-96c351a0e591-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:43:53.780365 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.780318    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-home" (OuterVolumeSpecName: "home") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:53.781721 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.781692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kube-api-access-5c57z" (OuterVolumeSpecName: "kube-api-access-5c57z") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "kube-api-access-5c57z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:43:53.782010 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.781983    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:43:53.782108 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.782084    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-dshm" (OuterVolumeSpecName: "dshm") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:53.797573 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.797538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:53.800775 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.800753    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f79415e-a7ea-4c32-a5b8-96c351a0e591" (UID: "3f79415e-a7ea-4c32-a5b8-96c351a0e591"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:43:53.880649 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880615    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880649 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880643    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880649 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880653    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880662    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c57z\" (UniqueName: \"kubernetes.io/projected/3f79415e-a7ea-4c32-a5b8-96c351a0e591-kube-api-access-5c57z\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880670    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880678    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880686    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f79415e-a7ea-4c32-a5b8-96c351a0e591-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:53.880882 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:53.880693    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f79415e-a7ea-4c32-a5b8-96c351a0e591-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:43:54.484755 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.484714    2560 generic.go:356] "Generic (PLEG): container finished" podID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerID="1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39" exitCode=0
Sep 16 17:43:54.485255 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.484769    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"
Sep 16 17:43:54.485255 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.484809    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" event={"ID":"3f79415e-a7ea-4c32-a5b8-96c351a0e591","Type":"ContainerDied","Data":"1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39"}
Sep 16 17:43:54.485255 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.484883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs" event={"ID":"3f79415e-a7ea-4c32-a5b8-96c351a0e591","Type":"ContainerDied","Data":"8c073e02439e4ee6c430a58d19872001a3ccadd355190825445680dfa364c8d3"}
Sep 16 17:43:54.485255 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.484919    2560 scope.go:117] "RemoveContainer" containerID="1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39"
Sep 16 17:43:54.494534 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.494516    2560 scope.go:117] "RemoveContainer" containerID="ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0"
Sep 16 17:43:54.501885 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.501869    2560 scope.go:117] "RemoveContainer" containerID="1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39"
Sep 16 17:43:54.502152 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:43:54.502136    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39\": container with ID starting with 1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39 not found: ID does not exist" containerID="1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39"
Sep 16 17:43:54.502223 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.502162    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39"} err="failed to get container status \"1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39\": rpc error: code = NotFound desc = could not find container \"1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39\": container with ID starting with 1c9b19e881fbf1819827e4b063d5de29c00c58558b7087f2870df4bd30a7fd39 not found: ID does not exist"
Sep 16 17:43:54.502223 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.502183    2560 scope.go:117] "RemoveContainer" containerID="ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0"
Sep 16 17:43:54.502424 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:43:54.502407    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0\": container with ID starting with ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0 not found: ID does not exist" containerID="ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0"
Sep 16 17:43:54.502462 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.502434    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0"} err="failed to get container status \"ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0\": rpc error: code = NotFound desc = could not find container \"ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0\": container with ID starting with ecc5a3ec23999e1e1a3a7c2fdf2755f2b5db8b4244428616eb4904610cf798a0 not found: ID does not exist"
Sep 16 17:43:54.511001 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.510977    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"]
Sep 16 17:43:54.519941 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:54.519921    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-fac1c4ae/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7b75db48cbc4qjs"]
Sep 16 17:43:55.117608 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:43:55.117542    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" path="/var/lib/kubelet/pods/3f79415e-a7ea-4c32-a5b8-96c351a0e591/volumes"
Sep 16 17:44:03.727499 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.727467    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 17:44:03.727892 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.727816    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main"
Sep 16 17:44:03.727892 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.727830    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main"
Sep 16 17:44:03.727892 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.727846    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="storage-initializer"
Sep 16 17:44:03.727892 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.727854    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="storage-initializer"
Sep 16 17:44:03.728051 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.727908    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f79415e-a7ea-4c32-a5b8-96c351a0e591" containerName="main"
Sep 16 17:44:03.732447 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.732429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.734729 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.734709    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 17:44:03.735601 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.735568    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-fzqtd\""
Sep 16 17:44:03.735719 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.735583    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 17:44:03.735719 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.735634    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 17:44:03.735719 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.735583    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-cmk64\""
Sep 16 17:44:03.735719 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.735584    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 16 17:44:03.736006 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.735989    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 17:44:03.749930 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.749910    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 17:44:03.863783 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.863754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgcl\" (UniqueName: \"kubernetes.io/projected/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kube-api-access-2cgcl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.863947 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.863802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.863947 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.863865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.863947 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.863919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.863947 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.863940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.864117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.863968    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.864117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.864015    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.864117 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.864032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965022 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.964989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgcl\" (UniqueName: \"kubernetes.io/projected/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kube-api-access-2cgcl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965022 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965228 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965228 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965228 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965400 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965456 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965456 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965456 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965348    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965456 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965658 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.965694 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.965671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.966030 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.966008    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.967367 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.967339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.967496 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.967390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:03.975345 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:03.975321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgcl\" (UniqueName: \"kubernetes.io/projected/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kube-api-access-2cgcl\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:04.042498 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.042431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:44:04.326401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.326371    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 17:44:04.328903 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:44:04.328870    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560a5586_0d42_4c9f_8a63_291e5c5ae5a1.slice/crio-ecd0b6db4955afee04b9ea52082450b6b15659410a24840a2f4f5175b5766a9b WatchSource:0}: Error finding container ecd0b6db4955afee04b9ea52082450b6b15659410a24840a2f4f5175b5766a9b: Status 404 returned error can't find the container with id ecd0b6db4955afee04b9ea52082450b6b15659410a24840a2f4f5175b5766a9b
Sep 16 17:44:04.528991 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.528945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"560a5586-0d42-4c9f-8a63-291e5c5ae5a1","Type":"ContainerStarted","Data":"e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5"}
Sep 16 17:44:04.529171 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.529001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"560a5586-0d42-4c9f-8a63-291e5c5ae5a1","Type":"ContainerStarted","Data":"ecd0b6db4955afee04b9ea52082450b6b15659410a24840a2f4f5175b5766a9b"}
Sep 16 17:44:04.687603 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.687553    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"]
Sep 16 17:44:04.691286 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.691260    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.697564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.697537    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-pp9z4\""
Sep 16 17:44:04.697720 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.697679    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Sep 16 17:44:04.698067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.698046    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Sep 16 17:44:04.698526 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.698508    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Sep 16 17:44:04.712852 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.712793    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"]
Sep 16 17:44:04.772201 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772169    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772201 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x22z\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-kube-api-access-8x22z\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.772715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.772611    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873136 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x22z\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-kube-api-access-8x22z\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873626 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873626 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873626 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873626 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873550    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-credential-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873626 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-data\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.873884 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-socket\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.874034 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.873998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-certs\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.874231 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.874210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.876333 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.876303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-podinfo\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.876527 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.876509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-envoy\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.886883 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.886851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-token\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:04.887799 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:04.887773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x22z\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-kube-api-access-8x22z\") pod \"router-gateway-1-openshift-default-5b4d9458bd-2np7q\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:05.002283 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.002247    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:05.285732 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.285662    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"]
Sep 16 17:44:05.289177 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:44:05.289150    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe84d5d3_ea8e_45b7_93ed_dbaa5a7c5948.slice/crio-90b3f97055d6b066f5c688ea91e28c8c8b99564e80de6eea5afacca5e3c9acdb WatchSource:0}: Error finding container 90b3f97055d6b066f5c688ea91e28c8c8b99564e80de6eea5afacca5e3c9acdb: Status 404 returned error can't find the container with id 90b3f97055d6b066f5c688ea91e28c8c8b99564e80de6eea5afacca5e3c9acdb
Sep 16 17:44:05.291937 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.291900    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:44:05.292003 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.291992    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:44:05.292054 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.292040    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:44:05.536837 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.536740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q" event={"ID":"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948","Type":"ContainerStarted","Data":"26124c26977fa9711bcbb77101e2505b80b8879d20f6a0d931e0575874dda2fe"}
Sep 16 17:44:05.536837 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.536814    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q" event={"ID":"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948","Type":"ContainerStarted","Data":"90b3f97055d6b066f5c688ea91e28c8c8b99564e80de6eea5afacca5e3c9acdb"}
Sep 16 17:44:05.558687 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:05.558640    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q" podStartSLOduration=1.558626017 podStartE2EDuration="1.558626017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:05.556607049 +0000 UTC m=+1160.965724297" watchObservedRunningTime="2026-09-16 17:44:05.558626017 +0000 UTC m=+1160.967743265"
Sep 16 17:44:06.003396 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:06.003355    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:06.008607 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:06.008562    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:06.541424 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:06.541383    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:06.542331 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:06.542309    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:07.546896 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:07.546817    2560 generic.go:356] "Generic (PLEG): container finished" podID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerID="e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5" exitCode=0
Sep 16 17:44:07.547266 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:07.546898    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"560a5586-0d42-4c9f-8a63-291e5c5ae5a1","Type":"ContainerDied","Data":"e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5"}
Sep 16 17:44:08.552805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:08.552767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"560a5586-0d42-4c9f-8a63-291e5c5ae5a1","Type":"ContainerStarted","Data":"de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9"}
Sep 16 17:44:08.579423 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:08.579372    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.250064867 podStartE2EDuration="5.579357201s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.329292334s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:08.578055698 +0000 UTC m=+1163.987172946" watchObservedRunningTime="2026-09-16 17:44:08.579357201 +0000 UTC m=+1163.988474448"
Sep 16 17:44:43.577400 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.577365    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"]
Sep 16 17:44:43.579566 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.579549    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.582111 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.582084    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 16 17:44:43.582558 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.582540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-md2k4\""
Sep 16 17:44:43.582695 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.582571    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 16 17:44:43.582850 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.582830    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 16 17:44:43.596838 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.596810    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"]
Sep 16 17:44:43.718029 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.717983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9gbc\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-kube-api-access-n9gbc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718459 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718459 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718459 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.718459 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.718342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819030 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.818980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9gbc\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-kube-api-access-n9gbc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819137    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819577 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819242    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819577 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.819814 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.819785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-certs\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.820093 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.820067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-data\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.820710 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.820683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.821221 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.821191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-credential-socket\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.822310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.822247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.825608 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.824467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-envoy\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.826617 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.826554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-podinfo\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.830608 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.829106    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9gbc\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-kube-api-access-n9gbc\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.831034 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.831006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-token\") pod \"router-gateway-1-openshift-default-66bbccc5fb-l7xt7\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:43.890220 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:43.890185    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:44.192258 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.192061    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"]
Sep 16 17:44:44.195648 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:44:44.195616    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a9f9a0_55fe_41c3_ab68_814ae91434bc.slice/crio-432e49f47d664aac02308f775e9e5803f87e7b021a976e5c102ca4be0d6584a1 WatchSource:0}: Error finding container 432e49f47d664aac02308f775e9e5803f87e7b021a976e5c102ca4be0d6584a1: Status 404 returned error can't find the container with id 432e49f47d664aac02308f775e9e5803f87e7b021a976e5c102ca4be0d6584a1
Sep 16 17:44:44.197647 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.197618    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:44:44.197740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.197682    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:44:44.197740 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.197712    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 16 17:44:44.710390 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.710351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7" event={"ID":"18a9f9a0-55fe-41c3-ab68-814ae91434bc","Type":"ContainerStarted","Data":"4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f"}
Sep 16 17:44:44.710874 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.710404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7" event={"ID":"18a9f9a0-55fe-41c3-ab68-814ae91434bc","Type":"ContainerStarted","Data":"432e49f47d664aac02308f775e9e5803f87e7b021a976e5c102ca4be0d6584a1"}
Sep 16 17:44:44.891283 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:44.891250    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:45.896041 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:45.896008    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:45.926037 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:45.925965    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7" podStartSLOduration=2.925943354 podStartE2EDuration="2.925943354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:44:44.739839711 +0000 UTC m=+1200.148956961" watchObservedRunningTime="2026-09-16 17:44:45.925943354 +0000 UTC m=+1201.335060605"
Sep 16 17:44:46.721584 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:46.721541    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:46.727175 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:46.727133    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:44:48.259160 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:48.259124    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"]
Sep 16 17:44:48.259559 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:48.259333    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q" podUID="be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" containerName="istio-proxy" containerID="cri-o://26124c26977fa9711bcbb77101e2505b80b8879d20f6a0d931e0575874dda2fe" gracePeriod=30
Sep 16 17:44:50.742715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.742676    2560 generic.go:356] "Generic (PLEG): container finished" podID="be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" containerID="26124c26977fa9711bcbb77101e2505b80b8879d20f6a0d931e0575874dda2fe" exitCode=0
Sep 16 17:44:50.743066 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.742737    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q" event={"ID":"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948","Type":"ContainerDied","Data":"26124c26977fa9711bcbb77101e2505b80b8879d20f6a0d931e0575874dda2fe"}
Sep 16 17:44:50.798499 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.798446    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:50.904632 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-envoy\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904764 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904682    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istiod-ca-cert\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904764 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-certs\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904764 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904731    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-socket\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904917 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x22z\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-kube-api-access-8x22z\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904917 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904812    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-credential-socket\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904917 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904845    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-data\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.904917 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904869    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-token\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.905109 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.904924    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-podinfo\") pod \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\" (UID: \"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948\") "
Sep 16 17:44:50.905109 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905040    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:50.905221 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905104    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:44:50.905221 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905160    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-data" (OuterVolumeSpecName: "istio-data") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:50.905332 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905226    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istiod-ca-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:50.905332 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905245    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:50.905431 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905372    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:50.905605 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.905535    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:50.907106 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.907071    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-kube-api-access-8x22z" (OuterVolumeSpecName: "kube-api-access-8x22z") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "kube-api-access-8x22z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:44:50.907218 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.907184    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:44:50.907218 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.907200    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-token" (OuterVolumeSpecName: "istio-token") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:44:50.907346 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:50.907321    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" (UID: "be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:44:51.006328 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006290    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-workload-socket\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.006328 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006328    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x22z\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-kube-api-access-8x22z\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.006514 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006345    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-credential-socket\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.006514 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006360    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-data\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.006514 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006374    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-token\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.006514 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006386    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-podinfo\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.006514 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.006398    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948-istio-envoy\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:44:51.749491 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.749460    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"
Sep 16 17:44:51.749959 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.749489    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q" event={"ID":"be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948","Type":"ContainerDied","Data":"90b3f97055d6b066f5c688ea91e28c8c8b99564e80de6eea5afacca5e3c9acdb"}
Sep 16 17:44:51.749959 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.749541    2560 scope.go:117] "RemoveContainer" containerID="26124c26977fa9711bcbb77101e2505b80b8879d20f6a0d931e0575874dda2fe"
Sep 16 17:44:51.765512 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.765490    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"]
Sep 16 17:44:51.769316 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:51.769295    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-5b4d9458bd-2np7q"]
Sep 16 17:44:53.117080 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:44:53.117050    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" path="/var/lib/kubelet/pods/be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948/volumes"
Sep 16 17:45:10.760715 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:10.760682    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"]
Sep 16 17:45:10.761102 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:10.760898    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7" podUID="18a9f9a0-55fe-41c3-ab68-814ae91434bc" containerName="istio-proxy" containerID="cri-o://4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f" gracePeriod=30
Sep 16 17:45:13.510248 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.510224    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:45:13.581571 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581491    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-podinfo\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.581571 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581550    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-data\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.581834 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581575    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-token\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.581834 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581641    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-credential-socket\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.581834 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581801    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-certs\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.582072 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-envoy\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.582072 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581905    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istiod-ca-cert\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.582072 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581943    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9gbc\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-kube-api-access-n9gbc\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.582072 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.581980    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-socket\") pod \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\" (UID: \"18a9f9a0-55fe-41c3-ab68-814ae91434bc\") "
Sep 16 17:45:13.582709 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.582651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:45:13.582830 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.582798    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-data" (OuterVolumeSpecName: "istio-data") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:13.583038 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.583007    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:13.584306 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.584275    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:45:13.585116 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.585065    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:13.585363 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.585336    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:13.585894 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.585822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:45:13.585992 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.585924    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-kube-api-access-n9gbc" (OuterVolumeSpecName: "kube-api-access-n9gbc") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "kube-api-access-n9gbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:13.587453 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.587424    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-token" (OuterVolumeSpecName: "istio-token") pod "18a9f9a0-55fe-41c3-ab68-814ae91434bc" (UID: "18a9f9a0-55fe-41c3-ab68-814ae91434bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:45:13.683547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683519    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istiod-ca-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683545    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9gbc\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-kube-api-access-n9gbc\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683547 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683554    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-socket\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683564    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-podinfo\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683572    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-data\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683582    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-token\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683604    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-credential-socket\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683612    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-workload-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.683844 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.683621    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a9f9a0-55fe-41c3-ab68-814ae91434bc-istio-envoy\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:45:13.862743 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.862710    2560 generic.go:356] "Generic (PLEG): container finished" podID="18a9f9a0-55fe-41c3-ab68-814ae91434bc" containerID="4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f" exitCode=0
Sep 16 17:45:13.862914 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.862774    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"
Sep 16 17:45:13.862914 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.862804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7" event={"ID":"18a9f9a0-55fe-41c3-ab68-814ae91434bc","Type":"ContainerDied","Data":"4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f"}
Sep 16 17:45:13.862914 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.862857    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7" event={"ID":"18a9f9a0-55fe-41c3-ab68-814ae91434bc","Type":"ContainerDied","Data":"432e49f47d664aac02308f775e9e5803f87e7b021a976e5c102ca4be0d6584a1"}
Sep 16 17:45:13.862914 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.862882    2560 scope.go:117] "RemoveContainer" containerID="4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f"
Sep 16 17:45:13.871082 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.871065    2560 scope.go:117] "RemoveContainer" containerID="4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f"
Sep 16 17:45:13.871357 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:45:13.871333    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f\": container with ID starting with 4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f not found: ID does not exist" containerID="4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f"
Sep 16 17:45:13.871445 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.871364    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f"} err="failed to get container status \"4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f\": rpc error: code = NotFound desc = could not find container \"4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f\": container with ID starting with 4e9efcdac31e1498a03e6b7e8d0a509771a7b7b1aa0087a08ef69c23fc38a01f not found: ID does not exist"
Sep 16 17:45:13.889917 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.889897    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"]
Sep 16 17:45:13.893238 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:13.893218    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-66bbccc5fb-l7xt7"]
Sep 16 17:45:15.119346 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.119300    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a9f9a0-55fe-41c3-ab68-814ae91434bc" path="/var/lib/kubelet/pods/18a9f9a0-55fe-41c3-ab68-814ae91434bc/volumes"
Sep 16 17:45:15.134707 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.134674    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"]
Sep 16 17:45:15.134965 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.134953    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" containerName="istio-proxy"
Sep 16 17:45:15.135006 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.134967    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" containerName="istio-proxy"
Sep 16 17:45:15.135006 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.134984    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a9f9a0-55fe-41c3-ab68-814ae91434bc" containerName="istio-proxy"
Sep 16 17:45:15.135006 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.134990    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a9f9a0-55fe-41c3-ab68-814ae91434bc" containerName="istio-proxy"
Sep 16 17:45:15.135099 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.135047    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be84d5d3-ea8e-45b7-93ed-dbaa5a7c5948" containerName="istio-proxy"
Sep 16 17:45:15.135099 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.135055    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a9f9a0-55fe-41c3-ab68-814ae91434bc" containerName="istio-proxy"
Sep 16 17:45:15.137576 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.137560    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.140325 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.140255    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 17:45:15.140325 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.140278    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-8x5nr\""
Sep 16 17:45:15.140325 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.140276    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 17:45:15.140577 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.140359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-n6ttx\""
Sep 16 17:45:15.140577 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.140410    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 17:45:15.147401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.147377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"]
Sep 16 17:45:15.196339 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.196301    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.196502 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.196356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p228z\" (UniqueName: \"kubernetes.io/projected/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-kube-api-access-p228z\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.297425 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.297385    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p228z\" (UniqueName: \"kubernetes.io/projected/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-kube-api-access-p228z\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.297639 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.297445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.299977 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.299945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.305759 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.305728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p228z\" (UniqueName: \"kubernetes.io/projected/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-kube-api-access-p228z\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.449568 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.449479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:15.753465 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.753432    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"]
Sep 16 17:45:15.755797 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:45:15.755769    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93cf251_0034_48a5_9c20_e7d71a5bb4a7.slice/crio-0e2d09f3d4467711913368b1bfc4e511b37cbf8dc64a7356f4ce6a6308dc169c WatchSource:0}: Error finding container 0e2d09f3d4467711913368b1bfc4e511b37cbf8dc64a7356f4ce6a6308dc169c: Status 404 returned error can't find the container with id 0e2d09f3d4467711913368b1bfc4e511b37cbf8dc64a7356f4ce6a6308dc169c
Sep 16 17:45:15.873767 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:15.873732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" event={"ID":"e93cf251-0034-48a5-9c20-e7d71a5bb4a7","Type":"ContainerStarted","Data":"0e2d09f3d4467711913368b1bfc4e511b37cbf8dc64a7356f4ce6a6308dc169c"}
Sep 16 17:45:18.896769 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:18.896733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" event={"ID":"e93cf251-0034-48a5-9c20-e7d71a5bb4a7","Type":"ContainerStarted","Data":"99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8"}
Sep 16 17:45:18.897163 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:18.896870    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:45:18.915506 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:18.915453    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" podStartSLOduration=1.538859962 podStartE2EDuration="3.915434765s" totalImagesPullingTime="2.376574803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:45:18.914163996 +0000 UTC m=+1234.323281244" watchObservedRunningTime="2026-09-16 17:45:18.915434765 +0000 UTC m=+1234.324552013"
Sep 16 17:45:49.903318 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:45:49.903283    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:46:31.112429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:31.112388    2560 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-fzqtd\" not found"
Sep 16 17:46:31.216417 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:31.216382    2560 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:31.216628 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:31.216472    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs podName:560a5586-0d42-4c9f-8a63-291e5c5ae5a1 nodeName:}" failed. No retries permitted until 2026-09-16 17:46:31.716451124 +0000 UTC m=+1307.125568349 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:31.720764 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:31.720715    2560 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:31.720951 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:31.720799    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs podName:560a5586-0d42-4c9f-8a63-291e5c5ae5a1 nodeName:}" failed. No retries permitted until 2026-09-16 17:46:32.720784488 +0000 UTC m=+1308.129901714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:32.727568 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:32.727531    2560 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:32.727963 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:32.727634    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs podName:560a5586-0d42-4c9f-8a63-291e5c5ae5a1 nodeName:}" failed. No retries permitted until 2026-09-16 17:46:34.727616142 +0000 UTC m=+1310.136733369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:34.744464 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:34.744424    2560 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:34.744866 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:34.744506    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs podName:560a5586-0d42-4c9f-8a63-291e5c5ae5a1 nodeName:}" failed. No retries permitted until 2026-09-16 17:46:38.744491289 +0000 UTC m=+1314.153608515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:36.527298 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:36.527258    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 17:46:38.779037 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:38.778999    2560 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:38.779573 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:38.779108    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs podName:560a5586-0d42-4c9f-8a63-291e5c5ae5a1 nodeName:}" failed. No retries permitted until 2026-09-16 17:46:46.779087711 +0000 UTC m=+1322.188204953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:46.846040 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:46.846006    2560 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:46.846413 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:46.846090    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs podName:560a5586-0d42-4c9f-8a63-291e5c5ae5a1 nodeName:}" failed. No retries permitted until 2026-09-16 17:47:02.846071283 +0000 UTC m=+1338.255188517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Sep 16 17:46:51.548974 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:51.548917    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerName="main" containerID="cri-o://de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9" gracePeriod=45
Sep 16 17:46:52.197554 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.197529    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:46:52.291961 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.291879    2560 generic.go:356] "Generic (PLEG): container finished" podID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerID="de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9" exitCode=0
Sep 16 17:46:52.291961 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.291933    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 16 17:46:52.292125 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.291960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"560a5586-0d42-4c9f-8a63-291e5c5ae5a1","Type":"ContainerDied","Data":"de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9"}
Sep 16 17:46:52.292125 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292015    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"560a5586-0d42-4c9f-8a63-291e5c5ae5a1","Type":"ContainerDied","Data":"ecd0b6db4955afee04b9ea52082450b6b15659410a24840a2f4f5175b5766a9b"}
Sep 16 17:46:52.292125 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292039    2560 scope.go:117] "RemoveContainer" containerID="de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9"
Sep 16 17:46:52.292445 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-model-cache\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292534 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cgcl\" (UniqueName: \"kubernetes.io/projected/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kube-api-access-2cgcl\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292534 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292495    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-home\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292661 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292561    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tmp-dir\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292661 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292675    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-cabundle-cert\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kserve-provision-location\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-dshm\") pod \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\" (UID: \"560a5586-0d42-4c9f-8a63-291e5c5ae5a1\") "
Sep 16 17:46:52.292765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:52.293063 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.292987    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.293184 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.293100    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:46:52.293313 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.293279    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-home" (OuterVolumeSpecName: "home") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:52.294777 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.294754    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kube-api-access-2cgcl" (OuterVolumeSpecName: "kube-api-access-2cgcl") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "kube-api-access-2cgcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:46:52.295284 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.295260    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:46:52.295376 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.295294    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-dshm" (OuterVolumeSpecName: "dshm") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:52.310504 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.310479    2560 scope.go:117] "RemoveContainer" containerID="e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5"
Sep 16 17:46:52.310752 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.310717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:52.317933 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.317915    2560 scope.go:117] "RemoveContainer" containerID="de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9"
Sep 16 17:46:52.318200 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:52.318180    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9\": container with ID starting with de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9 not found: ID does not exist" containerID="de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9"
Sep 16 17:46:52.318261 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.318207    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9"} err="failed to get container status \"de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9\": rpc error: code = NotFound desc = could not find container \"de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9\": container with ID starting with de70513f4a0fb6ecb8e1eadee6bd31028bb7ea87b42364c8716d0ae34c8845f9 not found: ID does not exist"
Sep 16 17:46:52.318261 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.318224    2560 scope.go:117] "RemoveContainer" containerID="e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5"
Sep 16 17:46:52.318445 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:46:52.318429    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5\": container with ID starting with e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5 not found: ID does not exist" containerID="e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5"
Sep 16 17:46:52.318490 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.318449    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5"} err="failed to get container status \"e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5\": rpc error: code = NotFound desc = could not find container \"e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5\": container with ID starting with e0879430795a62cc10c7dec2bbdd0addee9adcc267fb5b03c2b68c236470cbc5 not found: ID does not exist"
Sep 16 17:46:52.321531 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.321509    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "560a5586-0d42-4c9f-8a63-291e5c5ae5a1" (UID: "560a5586-0d42-4c9f-8a63-291e5c5ae5a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:46:52.394051 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394023    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.394051 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394048    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cgcl\" (UniqueName: \"kubernetes.io/projected/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kube-api-access-2cgcl\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.394211 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394058    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.394211 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394066    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.394211 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394076    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.394211 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394086    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.394211 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.394094    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/560a5586-0d42-4c9f-8a63-291e5c5ae5a1-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:46:52.618940 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.618908    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 17:46:52.624970 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:52.624940    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 16 17:46:53.117695 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:46:53.117666    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" path="/var/lib/kubelet/pods/560a5586-0d42-4c9f-8a63-291e5c5ae5a1/volumes"
Sep 16 17:47:21.170696 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.170614    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"]
Sep 16 17:47:21.171151 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.170930    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerName="main"
Sep 16 17:47:21.171151 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.170942    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerName="main"
Sep 16 17:47:21.171151 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.170952    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerName="storage-initializer"
Sep 16 17:47:21.171151 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.170957    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerName="storage-initializer"
Sep 16 17:47:21.171151 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.171034    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="560a5586-0d42-4c9f-8a63-291e5c5ae5a1" containerName="main"
Sep 16 17:47:21.173806 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.173786    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.176578 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.176560    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 17:47:21.177195 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.177165    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 17:47:21.177275 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.177223    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 17:47:21.177275 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.177262    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-xrrq6\""
Sep 16 17:47:21.177356 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.177283    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-pd9ch\""
Sep 16 17:47:21.185283 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.185261    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"]
Sep 16 17:47:21.218942 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.218906    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.219060 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.218962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwcb9\" (UniqueName: \"kubernetes.io/projected/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-kube-api-access-fwcb9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.320379 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.320345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.320524 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.320396    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwcb9\" (UniqueName: \"kubernetes.io/projected/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-kube-api-access-fwcb9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.322767 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.322746    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.328333 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.328309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwcb9\" (UniqueName: \"kubernetes.io/projected/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-kube-api-access-fwcb9\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.483854 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.483827    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:21.777148 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:21.777063    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"]
Sep 16 17:47:21.780957 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:47:21.780926    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6922d6d9_bdf4_430b_b3db_44ec5fb67b30.slice/crio-7ebc47421dfdd2af5df6c451179b0f9da3e44358f4658664cb7d275002a35fd3 WatchSource:0}: Error finding container 7ebc47421dfdd2af5df6c451179b0f9da3e44358f4658664cb7d275002a35fd3: Status 404 returned error can't find the container with id 7ebc47421dfdd2af5df6c451179b0f9da3e44358f4658664cb7d275002a35fd3
Sep 16 17:47:22.415314 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:22.415279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" event={"ID":"6922d6d9-bdf4-430b-b3db-44ec5fb67b30","Type":"ContainerStarted","Data":"42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095"}
Sep 16 17:47:22.415803 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:22.415339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" event={"ID":"6922d6d9-bdf4-430b-b3db-44ec5fb67b30","Type":"ContainerStarted","Data":"7ebc47421dfdd2af5df6c451179b0f9da3e44358f4658664cb7d275002a35fd3"}
Sep 16 17:47:22.415803 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:22.415451    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:22.444275 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:22.444218    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" podStartSLOduration=1.444203165 podStartE2EDuration="1.444203165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:47:22.441727951 +0000 UTC m=+1357.850845199" watchObservedRunningTime="2026-09-16 17:47:22.444203165 +0000 UTC m=+1357.853320412"
Sep 16 17:47:42.613976 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:42.613940    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"]
Sep 16 17:47:49.902450 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:49.902404    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:47:53.421853 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:53.421822    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:47:55.641220 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:55.641187    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"]
Sep 16 17:47:57.634087 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:57.634021    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main" containerID="cri-o://99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8" gracePeriod=45
Sep 16 17:47:59.903319 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:47:59.903276    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:03.421765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:03.421723    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:09.903171 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:09.903131    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:09.903725 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:09.903241    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:48:10.663505 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:10.663452    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main" containerID="cri-o://42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095" gracePeriod=45
Sep 16 17:48:13.421389 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:13.421348    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:19.903345 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:19.903305    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:23.422051 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:23.422008    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:23.422441 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:23.422117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:48:27.887700 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:27.887676    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:48:27.942635 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:27.942537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p228z\" (UniqueName: \"kubernetes.io/projected/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-kube-api-access-p228z\") pod \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") "
Sep 16 17:48:27.942635 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:27.942622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-tls-certs\") pod \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\" (UID: \"e93cf251-0034-48a5-9c20-e7d71a5bb4a7\") "
Sep 16 17:48:27.944716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:27.944685    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-kube-api-access-p228z" (OuterVolumeSpecName: "kube-api-access-p228z") pod "e93cf251-0034-48a5-9c20-e7d71a5bb4a7" (UID: "e93cf251-0034-48a5-9c20-e7d71a5bb4a7"). InnerVolumeSpecName "kube-api-access-p228z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:48:27.944716 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:27.944701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e93cf251-0034-48a5-9c20-e7d71a5bb4a7" (UID: "e93cf251-0034-48a5-9c20-e7d71a5bb4a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:48:28.043288 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.043249    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:48:28.043288 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.043283    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p228z\" (UniqueName: \"kubernetes.io/projected/e93cf251-0034-48a5-9c20-e7d71a5bb4a7-kube-api-access-p228z\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:48:28.677512 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.677476    2560 generic.go:356] "Generic (PLEG): container finished" podID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerID="99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8" exitCode=0
Sep 16 17:48:28.677702 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.677520    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"
Sep 16 17:48:28.677702 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.677551    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" event={"ID":"e93cf251-0034-48a5-9c20-e7d71a5bb4a7","Type":"ContainerDied","Data":"99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8"}
Sep 16 17:48:28.677702 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.677609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl" event={"ID":"e93cf251-0034-48a5-9c20-e7d71a5bb4a7","Type":"ContainerDied","Data":"0e2d09f3d4467711913368b1bfc4e511b37cbf8dc64a7356f4ce6a6308dc169c"}
Sep 16 17:48:28.677702 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.677632    2560 scope.go:117] "RemoveContainer" containerID="99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8"
Sep 16 17:48:28.685723 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.685700    2560 scope.go:117] "RemoveContainer" containerID="99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8"
Sep 16 17:48:28.685989 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:48:28.685972    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8\": container with ID starting with 99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8 not found: ID does not exist" containerID="99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8"
Sep 16 17:48:28.686036 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.685997    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8"} err="failed to get container status \"99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8\": rpc error: code = NotFound desc = could not find container \"99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8\": container with ID starting with 99f5caa7128292b6bcefa96e5442a60da082df6187edf00f5c3ca11fc5450ca8 not found: ID does not exist"
Sep 16 17:48:28.708110 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.708085    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"]
Sep 16 17:48:28.717183 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:28.717162    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-2q9xl"]
Sep 16 17:48:29.117339 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:29.117297    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" path="/var/lib/kubelet/pods/e93cf251-0034-48a5-9c20-e7d71a5bb4a7/volumes"
Sep 16 17:48:33.421355 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:33.421317    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:48:35.596045 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.596014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"]
Sep 16 17:48:35.596436 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.596333    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main"
Sep 16 17:48:35.596436 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.596346    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main"
Sep 16 17:48:35.596436 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.596427    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e93cf251-0034-48a5-9c20-e7d71a5bb4a7" containerName="main"
Sep 16 17:48:35.599504 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.599483    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.606022 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.605993    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 16 17:48:35.606161 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.606053    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 16 17:48:35.606161 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.606068    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-2c8gk\""
Sep 16 17:48:35.606161 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.606061    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 16 17:48:35.606161 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.606153    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-sq2vf\""
Sep 16 17:48:35.622092 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.622068    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"]
Sep 16 17:48:35.693462 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.693431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxd52\" (UniqueName: \"kubernetes.io/projected/7533d793-4edc-42bf-ad81-5541453202ac-kube-api-access-wxd52\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.693645 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.693504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7533d793-4edc-42bf-ad81-5541453202ac-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.794075 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.794039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7533d793-4edc-42bf-ad81-5541453202ac-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.794269 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.794085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxd52\" (UniqueName: \"kubernetes.io/projected/7533d793-4edc-42bf-ad81-5541453202ac-kube-api-access-wxd52\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.796583 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.796557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7533d793-4edc-42bf-ad81-5541453202ac-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.801805 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.801785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxd52\" (UniqueName: \"kubernetes.io/projected/7533d793-4edc-42bf-ad81-5541453202ac-kube-api-access-wxd52\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:35.914745 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:35.914663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:36.195835 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:36.195751    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"]
Sep 16 17:48:36.198931 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:48:36.198892    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7533d793_4edc_42bf_ad81_5541453202ac.slice/crio-771d4e82402ad8dd6f892ab6a9bef79696c593563523d102625aad9b35c62bc4 WatchSource:0}: Error finding container 771d4e82402ad8dd6f892ab6a9bef79696c593563523d102625aad9b35c62bc4: Status 404 returned error can't find the container with id 771d4e82402ad8dd6f892ab6a9bef79696c593563523d102625aad9b35c62bc4
Sep 16 17:48:36.709401 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:36.709366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" event={"ID":"7533d793-4edc-42bf-ad81-5541453202ac","Type":"ContainerStarted","Data":"fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160"}
Sep 16 17:48:36.709824 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:36.709415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" event={"ID":"7533d793-4edc-42bf-ad81-5541453202ac","Type":"ContainerStarted","Data":"771d4e82402ad8dd6f892ab6a9bef79696c593563523d102625aad9b35c62bc4"}
Sep 16 17:48:36.709824 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:36.709494    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:48:36.733451 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:36.733411    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podStartSLOduration=1.7333847599999999 podStartE2EDuration="1.73338476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:36.733156666 +0000 UTC m=+1432.142273914" watchObservedRunningTime="2026-09-16 17:48:36.73338476 +0000 UTC m=+1432.142502009"
Sep 16 17:48:40.910217 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:40.910193    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:48:40.930429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:40.930361    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwcb9\" (UniqueName: \"kubernetes.io/projected/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-kube-api-access-fwcb9\") pod \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") "
Sep 16 17:48:40.930429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:40.930419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-tls-certs\") pod \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\" (UID: \"6922d6d9-bdf4-430b-b3db-44ec5fb67b30\") "
Sep 16 17:48:40.932515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:40.932492    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-kube-api-access-fwcb9" (OuterVolumeSpecName: "kube-api-access-fwcb9") pod "6922d6d9-bdf4-430b-b3db-44ec5fb67b30" (UID: "6922d6d9-bdf4-430b-b3db-44ec5fb67b30"). InnerVolumeSpecName "kube-api-access-fwcb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:48:40.932515 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:40.932506    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6922d6d9-bdf4-430b-b3db-44ec5fb67b30" (UID: "6922d6d9-bdf4-430b-b3db-44ec5fb67b30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:48:41.031758 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.031719    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwcb9\" (UniqueName: \"kubernetes.io/projected/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-kube-api-access-fwcb9\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:48:41.031758 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.031748    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6922d6d9-bdf4-430b-b3db-44ec5fb67b30-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:48:41.731226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.731189    2560 generic.go:356] "Generic (PLEG): container finished" podID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerID="42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095" exitCode=0
Sep 16 17:48:41.731412 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.731248    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"
Sep 16 17:48:41.731412 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.731252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" event={"ID":"6922d6d9-bdf4-430b-b3db-44ec5fb67b30","Type":"ContainerDied","Data":"42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095"}
Sep 16 17:48:41.731548 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.731411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz" event={"ID":"6922d6d9-bdf4-430b-b3db-44ec5fb67b30","Type":"ContainerDied","Data":"7ebc47421dfdd2af5df6c451179b0f9da3e44358f4658664cb7d275002a35fd3"}
Sep 16 17:48:41.731548 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.731445    2560 scope.go:117] "RemoveContainer" containerID="42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095"
Sep 16 17:48:41.739527 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.739503    2560 scope.go:117] "RemoveContainer" containerID="42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095"
Sep 16 17:48:41.739827 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:48:41.739803    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095\": container with ID starting with 42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095 not found: ID does not exist" containerID="42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095"
Sep 16 17:48:41.739905 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.739835    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095"} err="failed to get container status \"42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095\": rpc error: code = NotFound desc = could not find container \"42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095\": container with ID starting with 42b0a8352240e2166184854b4664d6573a6850670aa2d73954209d630b359095 not found: ID does not exist"
Sep 16 17:48:41.755743 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.755719    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"]
Sep 16 17:48:41.761330 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:41.761306    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbbftsz"]
Sep 16 17:48:43.117429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:43.117395    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" path="/var/lib/kubelet/pods/6922d6d9-bdf4-430b-b3db-44ec5fb67b30/volumes"
Sep 16 17:48:53.159895 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.159862    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"]
Sep 16 17:48:53.160279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.160148    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main"
Sep 16 17:48:53.160279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.160160    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main"
Sep 16 17:48:53.160279 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.160214    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6922d6d9-bdf4-430b-b3db-44ec5fb67b30" containerName="main"
Sep 16 17:48:53.162899 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.162882    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.178342 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.178319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 17:48:53.178443 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.178319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 17:48:53.178443 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.178319    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 17:48:53.178863 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.178849    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-5r7dk\""
Sep 16 17:48:53.179090 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.179076    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-l5vjz\""
Sep 16 17:48:53.190564 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.190544    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"]
Sep 16 17:48:53.227995 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.227964    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xtp\" (UniqueName: \"kubernetes.io/projected/87d378ca-1e33-41c3-b5c7-b981d6007ba7-kube-api-access-c9xtp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.228093 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.228006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d378ca-1e33-41c3-b5c7-b981d6007ba7-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.329202 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.329168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xtp\" (UniqueName: \"kubernetes.io/projected/87d378ca-1e33-41c3-b5c7-b981d6007ba7-kube-api-access-c9xtp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.329340 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.329232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d378ca-1e33-41c3-b5c7-b981d6007ba7-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.331605 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.331564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d378ca-1e33-41c3-b5c7-b981d6007ba7-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.353277 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.353235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xtp\" (UniqueName: \"kubernetes.io/projected/87d378ca-1e33-41c3-b5c7-b981d6007ba7-kube-api-access-c9xtp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.472475 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.472440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:53.769968 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.769880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"]
Sep 16 17:48:53.773388 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:48:53.773354    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d378ca_1e33_41c3_b5c7_b981d6007ba7.slice/crio-aad8810c2ed4a059af5260d34d9262d419f312b5b583875dbd0cb664c267b2f6 WatchSource:0}: Error finding container aad8810c2ed4a059af5260d34d9262d419f312b5b583875dbd0cb664c267b2f6: Status 404 returned error can't find the container with id aad8810c2ed4a059af5260d34d9262d419f312b5b583875dbd0cb664c267b2f6
Sep 16 17:48:53.779172 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:53.779121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" event={"ID":"87d378ca-1e33-41c3-b5c7-b981d6007ba7","Type":"ContainerStarted","Data":"aad8810c2ed4a059af5260d34d9262d419f312b5b583875dbd0cb664c267b2f6"}
Sep 16 17:48:54.784089 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:54.784055    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" event={"ID":"87d378ca-1e33-41c3-b5c7-b981d6007ba7","Type":"ContainerStarted","Data":"285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d"}
Sep 16 17:48:54.784480 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:54.784167    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:48:54.807200 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:48:54.807158    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" podStartSLOduration=1.8071406300000001 podStartE2EDuration="1.80714063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:48:54.805362793 +0000 UTC m=+1450.214480041" watchObservedRunningTime="2026-09-16 17:48:54.80714063 +0000 UTC m=+1450.216257879"
Sep 16 17:49:07.715376 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:07.715338    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:49:25.790034 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:25.790006    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:49:27.618637 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:27.618582    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"]
Sep 16 17:49:35.789650 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:35.789608    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:49:42.637255 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:42.637183    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main" containerID="cri-o://285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d" gracePeriod=45
Sep 16 17:49:45.789861 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:45.789819    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:49:55.790481 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:55.790445    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:49:55.790899 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:49:55.790547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:50:05.790333 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:05.790296    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:50:12.874876 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:12.874845    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:50:12.973183 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:12.973152    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9xtp\" (UniqueName: \"kubernetes.io/projected/87d378ca-1e33-41c3-b5c7-b981d6007ba7-kube-api-access-c9xtp\") pod \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") "
Sep 16 17:50:12.973337 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:12.973213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d378ca-1e33-41c3-b5c7-b981d6007ba7-tls-certs\") pod \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\" (UID: \"87d378ca-1e33-41c3-b5c7-b981d6007ba7\") "
Sep 16 17:50:12.975388 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:12.975360    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d378ca-1e33-41c3-b5c7-b981d6007ba7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87d378ca-1e33-41c3-b5c7-b981d6007ba7" (UID: "87d378ca-1e33-41c3-b5c7-b981d6007ba7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:50:12.975498 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:12.975415    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d378ca-1e33-41c3-b5c7-b981d6007ba7-kube-api-access-c9xtp" (OuterVolumeSpecName: "kube-api-access-c9xtp") pod "87d378ca-1e33-41c3-b5c7-b981d6007ba7" (UID: "87d378ca-1e33-41c3-b5c7-b981d6007ba7"). InnerVolumeSpecName "kube-api-access-c9xtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:50:13.073959 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.073923    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xtp\" (UniqueName: \"kubernetes.io/projected/87d378ca-1e33-41c3-b5c7-b981d6007ba7-kube-api-access-c9xtp\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.073959 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.073953    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d378ca-1e33-41c3-b5c7-b981d6007ba7-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:50:13.099386 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.099351    2560 generic.go:356] "Generic (PLEG): container finished" podID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerID="285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d" exitCode=0
Sep 16 17:50:13.099530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.099397    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"
Sep 16 17:50:13.099530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.099446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" event={"ID":"87d378ca-1e33-41c3-b5c7-b981d6007ba7","Type":"ContainerDied","Data":"285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d"}
Sep 16 17:50:13.099530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.099491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k" event={"ID":"87d378ca-1e33-41c3-b5c7-b981d6007ba7","Type":"ContainerDied","Data":"aad8810c2ed4a059af5260d34d9262d419f312b5b583875dbd0cb664c267b2f6"}
Sep 16 17:50:13.099530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.099518    2560 scope.go:117] "RemoveContainer" containerID="285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d"
Sep 16 17:50:13.107971 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.107954    2560 scope.go:117] "RemoveContainer" containerID="285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d"
Sep 16 17:50:13.108275 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:50:13.108256    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d\": container with ID starting with 285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d not found: ID does not exist" containerID="285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d"
Sep 16 17:50:13.108332 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.108284    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d"} err="failed to get container status \"285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d\": rpc error: code = NotFound desc = could not find container \"285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d\": container with ID starting with 285253183ae77a3b4cbccffc8b731a4d8868f2ad5aaa117988eaf3b8b0f6248d not found: ID does not exist"
Sep 16 17:50:13.121273 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.121250    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"]
Sep 16 17:50:13.126104 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:13.126076    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schemcq9k"]
Sep 16 17:50:15.116417 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:15.116383    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" path="/var/lib/kubelet/pods/87d378ca-1e33-41c3-b5c7-b981d6007ba7/volumes"
Sep 16 17:50:22.390512 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.390473    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"]
Sep 16 17:50:22.391005 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.390811    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main"
Sep 16 17:50:22.391005 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.390824    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main"
Sep 16 17:50:22.391005 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.390881    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d378ca-1e33-41c3-b5c7-b981d6007ba7" containerName="main"
Sep 16 17:50:22.393524 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.393505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.396026 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.396004    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 17:50:22.396699 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.396680    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-bxcw4\""
Sep 16 17:50:22.396794 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.396722    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-nrqzn\""
Sep 16 17:50:22.396794 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.396736    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 17:50:22.396899 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.396827    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 17:50:22.415958 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.415935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"]
Sep 16 17:50:22.558423 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.558388    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5fx6\" (UniqueName: \"kubernetes.io/projected/2e7fd105-942a-4005-a063-f91b4ea09132-kube-api-access-z5fx6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.558607 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.558456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7fd105-942a-4005-a063-f91b4ea09132-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.659323 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.659234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7fd105-942a-4005-a063-f91b4ea09132-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.659478 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.659330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5fx6\" (UniqueName: \"kubernetes.io/projected/2e7fd105-942a-4005-a063-f91b4ea09132-kube-api-access-z5fx6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.661782 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.661760    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7fd105-942a-4005-a063-f91b4ea09132-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.667582 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.667557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5fx6\" (UniqueName: \"kubernetes.io/projected/2e7fd105-942a-4005-a063-f91b4ea09132-kube-api-access-z5fx6\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.705267 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.705240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:22.983625 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:22.983573    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"]
Sep 16 17:50:22.987390 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:50:22.987364    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7fd105_942a_4005_a063_f91b4ea09132.slice/crio-cf411c70da28baa2657509b11b2e2f1a9468772765379483cb29ad3000d520d0 WatchSource:0}: Error finding container cf411c70da28baa2657509b11b2e2f1a9468772765379483cb29ad3000d520d0: Status 404 returned error can't find the container with id cf411c70da28baa2657509b11b2e2f1a9468772765379483cb29ad3000d520d0
Sep 16 17:50:23.141255 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:23.141217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" event={"ID":"2e7fd105-942a-4005-a063-f91b4ea09132","Type":"ContainerStarted","Data":"aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a"}
Sep 16 17:50:23.141427 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:23.141270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" event={"ID":"2e7fd105-942a-4005-a063-f91b4ea09132","Type":"ContainerStarted","Data":"cf411c70da28baa2657509b11b2e2f1a9468772765379483cb29ad3000d520d0"}
Sep 16 17:50:23.141427 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:23.141291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:23.161939 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:23.161892    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" podStartSLOduration=1.1618737399999999 podStartE2EDuration="1.16187374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:50:23.160000596 +0000 UTC m=+1538.569117843" watchObservedRunningTime="2026-09-16 17:50:23.16187374 +0000 UTC m=+1538.570990988"
Sep 16 17:50:37.290967 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:37.290929    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"]
Sep 16 17:50:37.715920 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:37.715876    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:50:47.715238 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:47.715197    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:50:52.309747 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:52.309679    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main" containerID="cri-o://fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160" gracePeriod=45
Sep 16 17:50:54.148229 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:54.148188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:50:56.057247 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:56.057216    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"]
Sep 16 17:50:57.716110 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:57.716066    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:50:57.716520 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:50:57.716196    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:51:04.148233 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:04.148180    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:51:07.715500 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:07.715462    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:51:11.078245 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:11.078189    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main" containerID="cri-o://aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a" gracePeriod=45
Sep 16 17:51:14.148239 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:14.148198    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:51:17.715276 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:17.715231    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:51:22.548493 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.548469    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:51:22.638604 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.638515    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxd52\" (UniqueName: \"kubernetes.io/projected/7533d793-4edc-42bf-ad81-5541453202ac-kube-api-access-wxd52\") pod \"7533d793-4edc-42bf-ad81-5541453202ac\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") "
Sep 16 17:51:22.640577 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.640544    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7533d793-4edc-42bf-ad81-5541453202ac-kube-api-access-wxd52" (OuterVolumeSpecName: "kube-api-access-wxd52") pod "7533d793-4edc-42bf-ad81-5541453202ac" (UID: "7533d793-4edc-42bf-ad81-5541453202ac"). InnerVolumeSpecName "kube-api-access-wxd52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:51:22.739197 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.739171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7533d793-4edc-42bf-ad81-5541453202ac-tls-certs\") pod \"7533d793-4edc-42bf-ad81-5541453202ac\" (UID: \"7533d793-4edc-42bf-ad81-5541453202ac\") "
Sep 16 17:51:22.740493 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.739727    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxd52\" (UniqueName: \"kubernetes.io/projected/7533d793-4edc-42bf-ad81-5541453202ac-kube-api-access-wxd52\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:51:22.744610 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.741991    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7533d793-4edc-42bf-ad81-5541453202ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7533d793-4edc-42bf-ad81-5541453202ac" (UID: "7533d793-4edc-42bf-ad81-5541453202ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:51:22.840747 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:22.840723    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7533d793-4edc-42bf-ad81-5541453202ac-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:51:23.377530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.377447    2560 generic.go:356] "Generic (PLEG): container finished" podID="7533d793-4edc-42bf-ad81-5541453202ac" containerID="fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160" exitCode=0
Sep 16 17:51:23.377530 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.377503    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"
Sep 16 17:51:23.377737 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.377525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" event={"ID":"7533d793-4edc-42bf-ad81-5541453202ac","Type":"ContainerDied","Data":"fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160"}
Sep 16 17:51:23.377737 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.377579    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch" event={"ID":"7533d793-4edc-42bf-ad81-5541453202ac","Type":"ContainerDied","Data":"771d4e82402ad8dd6f892ab6a9bef79696c593563523d102625aad9b35c62bc4"}
Sep 16 17:51:23.377737 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.377614    2560 scope.go:117] "RemoveContainer" containerID="fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160"
Sep 16 17:51:23.385442 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.385422    2560 scope.go:117] "RemoveContainer" containerID="fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160"
Sep 16 17:51:23.385731 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:51:23.385704    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160\": container with ID starting with fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160 not found: ID does not exist" containerID="fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160"
Sep 16 17:51:23.385785 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.385740    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160"} err="failed to get container status \"fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160\": rpc error: code = NotFound desc = could not find container \"fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160\": container with ID starting with fb58398d3da555a86db50424f7c08ae7f0fd077bf424fe731106438abf3ea160 not found: ID does not exist"
Sep 16 17:51:23.400629 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.400606    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"]
Sep 16 17:51:23.409203 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:23.409181    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cwbcch"]
Sep 16 17:51:24.147635 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:24.147585    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:51:24.148081 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:24.147691    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:51:25.117446 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:25.117413    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7533d793-4edc-42bf-ad81-5541453202ac" path="/var/lib/kubelet/pods/7533d793-4edc-42bf-ad81-5541453202ac/volumes"
Sep 16 17:51:34.147974 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:34.147936    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:51:41.324149 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.324120    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:51:41.382327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.382244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7fd105-942a-4005-a063-f91b4ea09132-tls-certs\") pod \"2e7fd105-942a-4005-a063-f91b4ea09132\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") "
Sep 16 17:51:41.382460 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.382337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5fx6\" (UniqueName: \"kubernetes.io/projected/2e7fd105-942a-4005-a063-f91b4ea09132-kube-api-access-z5fx6\") pod \"2e7fd105-942a-4005-a063-f91b4ea09132\" (UID: \"2e7fd105-942a-4005-a063-f91b4ea09132\") "
Sep 16 17:51:41.384522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.384491    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7fd105-942a-4005-a063-f91b4ea09132-kube-api-access-z5fx6" (OuterVolumeSpecName: "kube-api-access-z5fx6") pod "2e7fd105-942a-4005-a063-f91b4ea09132" (UID: "2e7fd105-942a-4005-a063-f91b4ea09132"). InnerVolumeSpecName "kube-api-access-z5fx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:51:41.384522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.384514    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7fd105-942a-4005-a063-f91b4ea09132-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2e7fd105-942a-4005-a063-f91b4ea09132" (UID: "2e7fd105-942a-4005-a063-f91b4ea09132"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:51:41.445500 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.445459    2560 generic.go:356] "Generic (PLEG): container finished" podID="2e7fd105-942a-4005-a063-f91b4ea09132" containerID="aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a" exitCode=0
Sep 16 17:51:41.445681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.445506    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"
Sep 16 17:51:41.445681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.445514    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" event={"ID":"2e7fd105-942a-4005-a063-f91b4ea09132","Type":"ContainerDied","Data":"aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a"}
Sep 16 17:51:41.445681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.445567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs" event={"ID":"2e7fd105-942a-4005-a063-f91b4ea09132","Type":"ContainerDied","Data":"cf411c70da28baa2657509b11b2e2f1a9468772765379483cb29ad3000d520d0"}
Sep 16 17:51:41.445681 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.445617    2560 scope.go:117] "RemoveContainer" containerID="aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a"
Sep 16 17:51:41.453810 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.453792    2560 scope.go:117] "RemoveContainer" containerID="aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a"
Sep 16 17:51:41.454101 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:51:41.454082    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a\": container with ID starting with aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a not found: ID does not exist" containerID="aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a"
Sep 16 17:51:41.454153 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.454110    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a"} err="failed to get container status \"aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a\": rpc error: code = NotFound desc = could not find container \"aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a\": container with ID starting with aec2ed16b88c910ab9b62de927c77da10d40dd67cca36e5fada5f0fbce580b5a not found: ID does not exist"
Sep 16 17:51:41.465165 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.465134    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"]
Sep 16 17:51:41.469223 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.469199    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9s28cs"]
Sep 16 17:51:41.483259 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.483238    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7fd105-942a-4005-a063-f91b4ea09132-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:51:41.483259 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:41.483256    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5fx6\" (UniqueName: \"kubernetes.io/projected/2e7fd105-942a-4005-a063-f91b4ea09132-kube-api-access-z5fx6\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:51:43.117656 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:43.117613    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" path="/var/lib/kubelet/pods/2e7fd105-942a-4005-a063-f91b4ea09132/volumes"
Sep 16 17:51:54.137226 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137194    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"]
Sep 16 17:51:54.137668 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137468    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main"
Sep 16 17:51:54.137668 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137479    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main"
Sep 16 17:51:54.137668 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137491    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main"
Sep 16 17:51:54.137668 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137497    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main"
Sep 16 17:51:54.137668 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137553    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7533d793-4edc-42bf-ad81-5541453202ac" containerName="main"
Sep 16 17:51:54.137668 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.137560    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7fd105-942a-4005-a063-f91b4ea09132" containerName="main"
Sep 16 17:51:54.140463 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.140447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.144103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.144070    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 17:51:54.144103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.144091    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 17:51:54.144301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.144097    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-sqbws\""
Sep 16 17:51:54.144301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.144099    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-xsvwn\""
Sep 16 17:51:54.144301 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.144163    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 17:51:54.152630 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.152608    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"]
Sep 16 17:51:54.285048 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.285019    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a6b9993b-d59c-44e8-84f3-1f8663fe378b-kube-api-access-k9d5r\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.285048 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.285051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b9993b-d59c-44e8-84f3-1f8663fe378b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.386001 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.385974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a6b9993b-d59c-44e8-84f3-1f8663fe378b-kube-api-access-k9d5r\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.386127 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.386004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b9993b-d59c-44e8-84f3-1f8663fe378b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.388583 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.388536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b9993b-d59c-44e8-84f3-1f8663fe378b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.396057 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.396031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a6b9993b-d59c-44e8-84f3-1f8663fe378b-kube-api-access-k9d5r\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.450103 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.450077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:54.735265 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:54.728970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"]
Sep 16 17:51:55.505156 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:55.505121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" event={"ID":"a6b9993b-d59c-44e8-84f3-1f8663fe378b","Type":"ContainerStarted","Data":"3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944"}
Sep 16 17:51:55.505517 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:55.505180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" event={"ID":"a6b9993b-d59c-44e8-84f3-1f8663fe378b","Type":"ContainerStarted","Data":"7eb39684a48f918c6355250b21d1485554265b0783372c0d7c12dde772216613"}
Sep 16 17:51:55.505517 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:55.505308    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:51:55.562224 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:51:55.562169    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" podStartSLOduration=1.5621519130000001 podStartE2EDuration="1.562151913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:51:55.560113508 +0000 UTC m=+1630.969230756" watchObservedRunningTime="2026-09-16 17:51:55.562151913 +0000 UTC m=+1630.971269164"
Sep 16 17:52:26.511118 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:52:26.511086    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:53:58.837315 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:53:58.837281    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"]
Sep 16 17:54:06.510786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:06.510748    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:54:13.855726 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:13.855682    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main" containerID="cri-o://3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944" gracePeriod=45
Sep 16 17:54:16.510448 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:16.510411    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:54:26.510861 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:26.510822    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:54:26.511304 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:26.510934    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:54:36.511151 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:36.511106    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 17:54:44.103280 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.103255    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:54:44.184745 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.184668    2560 generic.go:356] "Generic (PLEG): container finished" podID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerID="3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944" exitCode=0
Sep 16 17:54:44.184745 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.184709    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"
Sep 16 17:54:44.184745 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.184716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" event={"ID":"a6b9993b-d59c-44e8-84f3-1f8663fe378b","Type":"ContainerDied","Data":"3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944"}
Sep 16 17:54:44.184968 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.184759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw" event={"ID":"a6b9993b-d59c-44e8-84f3-1f8663fe378b","Type":"ContainerDied","Data":"7eb39684a48f918c6355250b21d1485554265b0783372c0d7c12dde772216613"}
Sep 16 17:54:44.184968 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.184783    2560 scope.go:117] "RemoveContainer" containerID="3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944"
Sep 16 17:54:44.192631 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.192613    2560 scope.go:117] "RemoveContainer" containerID="3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944"
Sep 16 17:54:44.192895 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:54:44.192877    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944\": container with ID starting with 3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944 not found: ID does not exist" containerID="3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944"
Sep 16 17:54:44.192954 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.192902    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944"} err="failed to get container status \"3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944\": rpc error: code = NotFound desc = could not find container \"3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944\": container with ID starting with 3ad28b96a7de9e6900d134df8577b5cc4b7a688a1be67030f1da88f1cdc4b944 not found: ID does not exist"
Sep 16 17:54:44.198161 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.198143    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b9993b-d59c-44e8-84f3-1f8663fe378b-tls-certs\") pod \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") "
Sep 16 17:54:44.198218 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.198205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a6b9993b-d59c-44e8-84f3-1f8663fe378b-kube-api-access-k9d5r\") pod \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\" (UID: \"a6b9993b-d59c-44e8-84f3-1f8663fe378b\") "
Sep 16 17:54:44.200190 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.200165    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b9993b-d59c-44e8-84f3-1f8663fe378b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6b9993b-d59c-44e8-84f3-1f8663fe378b" (UID: "a6b9993b-d59c-44e8-84f3-1f8663fe378b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:54:44.200190 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.200180    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b9993b-d59c-44e8-84f3-1f8663fe378b-kube-api-access-k9d5r" (OuterVolumeSpecName: "kube-api-access-k9d5r") pod "a6b9993b-d59c-44e8-84f3-1f8663fe378b" (UID: "a6b9993b-d59c-44e8-84f3-1f8663fe378b"). InnerVolumeSpecName "kube-api-access-k9d5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:54:44.299362 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.299337    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6b9993b-d59c-44e8-84f3-1f8663fe378b-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:54:44.299362 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.299358    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9d5r\" (UniqueName: \"kubernetes.io/projected/a6b9993b-d59c-44e8-84f3-1f8663fe378b-kube-api-access-k9d5r\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:54:44.507627 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.507599    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"]
Sep 16 17:54:44.508925 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:44.508905    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-hsgdw"]
Sep 16 17:54:45.117402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:54:45.117366    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" path="/var/lib/kubelet/pods/a6b9993b-d59c-44e8-84f3-1f8663fe378b/volumes"
Sep 16 17:55:20.523749 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.523713    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"]
Sep 16 17:55:20.526061 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.524158    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main"
Sep 16 17:55:20.526061 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.524174    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main"
Sep 16 17:55:20.526061 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.524238    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b9993b-d59c-44e8-84f3-1f8663fe378b" containerName="main"
Sep 16 17:55:20.526833 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.526815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.531784 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.531741    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 16 17:55:20.531910 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.531834    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 16 17:55:20.531910 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.531742    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 16 17:55:20.531910 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.531776    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-vtflj\""
Sep 16 17:55:20.544041 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.544021    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"]
Sep 16 17:55:20.595231 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.595199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.595367 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.595236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.595367 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.595279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmvv\" (UniqueName: \"kubernetes.io/projected/a968e581-aa20-48da-9eb3-0fb43f863049-kube-api-access-5tmvv\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.595367 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.595309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.595511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.595384    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a968e581-aa20-48da-9eb3-0fb43f863049-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.595511 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.595457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696171 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696232    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmvv\" (UniqueName: \"kubernetes.io/projected/a968e581-aa20-48da-9eb3-0fb43f863049-kube-api-access-5tmvv\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696249    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696310 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a968e581-aa20-48da-9eb3-0fb43f863049-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696608 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696558    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696680 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696584    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.696680 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.696663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.698602 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.698568    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.698798 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.698781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a968e581-aa20-48da-9eb3-0fb43f863049-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.704419 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.704392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmvv\" (UniqueName: \"kubernetes.io/projected/a968e581-aa20-48da-9eb3-0fb43f863049-kube-api-access-5tmvv\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:20.836443 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:20.836365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:21.108104 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:21.108019    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"]
Sep 16 17:55:21.111334 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:55:21.111303    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda968e581_aa20_48da_9eb3_0fb43f863049.slice/crio-ca04784ed2be6fa53330f388b4ed2e36768e4db56e8d0eb2e2528b936a625fa4 WatchSource:0}: Error finding container ca04784ed2be6fa53330f388b4ed2e36768e4db56e8d0eb2e2528b936a625fa4: Status 404 returned error can't find the container with id ca04784ed2be6fa53330f388b4ed2e36768e4db56e8d0eb2e2528b936a625fa4
Sep 16 17:55:21.113426 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:21.113407    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:55:21.334135 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:21.334097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" event={"ID":"a968e581-aa20-48da-9eb3-0fb43f863049","Type":"ContainerStarted","Data":"ca04784ed2be6fa53330f388b4ed2e36768e4db56e8d0eb2e2528b936a625fa4"}
Sep 16 17:55:23.346399 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:23.346366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" event={"ID":"a968e581-aa20-48da-9eb3-0fb43f863049","Type":"ContainerStarted","Data":"011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"}
Sep 16 17:55:23.371512 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:23.371467    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" podStartSLOduration=1.744094142 podStartE2EDuration="3.371451895s" totalImagesPullingTime="1.627357753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:55:23.369776574 +0000 UTC m=+1838.778893833" watchObservedRunningTime="2026-09-16 17:55:23.371451895 +0000 UTC m=+1838.780569143"
Sep 16 17:55:23.836489 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:23.836459    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:30.837318 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:30.837281    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:30.849968 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:30.849946    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:55:31.390686 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:55:31.390654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:57:07.899192 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.899155    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"]
Sep 16 17:57:07.902176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.902159    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:07.905448 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.905413    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-n9fmz\""
Sep 16 17:57:07.905603 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.905521    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 17:57:07.905889 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.905874    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 17:57:07.906059 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.906045    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 17:57:07.928661 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:07.928634    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"]
Sep 16 17:57:08.051625 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.051571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.051625 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.051626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wt7d\" (UniqueName: \"kubernetes.io/projected/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-kube-api-access-8wt7d\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.051848 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.051693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-dshm\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.051848 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.051742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-home\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.051848 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.051761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.051848 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.051834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-model-cache\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152250 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-dshm\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152250 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152214    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-home\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152250 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152525 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-model-cache\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152525 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152302    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152525 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wt7d\" (UniqueName: \"kubernetes.io/projected/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-kube-api-access-8wt7d\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152741 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152713    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-model-cache\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152797 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.152797 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.152779    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-home\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.154475 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.154451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-dshm\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.154907 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.154887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tls-certs\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.162888 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.162865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wt7d\" (UniqueName: \"kubernetes.io/projected/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-kube-api-access-8wt7d\") pod \"test-div-v1-kserve-fb566d667-5hwwc\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.213355 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.213330    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:08.520195 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.520163    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"]
Sep 16 17:57:08.523420 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:57:08.523388    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719df1a8_82f6_4d3f_bca4_bfc0dea3319a.slice/crio-4d54e14fcf8bf6d55829ed472196b17d1c86bded5aa6070a0a90032ab90c1b35 WatchSource:0}: Error finding container 4d54e14fcf8bf6d55829ed472196b17d1c86bded5aa6070a0a90032ab90c1b35: Status 404 returned error can't find the container with id 4d54e14fcf8bf6d55829ed472196b17d1c86bded5aa6070a0a90032ab90c1b35
Sep 16 17:57:08.765722 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:08.765681    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc" event={"ID":"719df1a8-82f6-4d3f-bca4-bfc0dea3319a","Type":"ContainerStarted","Data":"4d54e14fcf8bf6d55829ed472196b17d1c86bded5aa6070a0a90032ab90c1b35"}
Sep 16 17:57:09.774092 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:09.774050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc" event={"ID":"719df1a8-82f6-4d3f-bca4-bfc0dea3319a","Type":"ContainerStarted","Data":"4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d"}
Sep 16 17:57:09.804418 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:09.804316    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc" podStartSLOduration=1.794255602 podStartE2EDuration="2.8042931s" totalImagesPullingTime="1.010037498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:57:09.803793908 +0000 UTC m=+1945.212911168" watchObservedRunningTime="2026-09-16 17:57:09.8042931 +0000 UTC m=+1945.213410348"
Sep 16 17:57:10.214035 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:10.214003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:18.214222 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:18.214187    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:18.226937 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:18.226916    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:18.819983 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:18.819952    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:57:24.934054 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:24.934020    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"]
Sep 16 17:57:24.942560 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:57:24.942509    2560 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:57:24.942560 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:57:24.942560 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 17:57:24.942560 ip-10-0-128-156 kubenswrapper[2560]:  > containerID="011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112" cmd=["/bin/sleep","15"]
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:24.942567    2560 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" message="" err=<
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]:  >
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:57:24.942578    2560 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" podUID="a968e581-aa20-48da-9eb3-0fb43f863049" containerName="main" containerID="cri-o://011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"
Sep 16 17:57:24.942831 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:24.942628    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" podUID="a968e581-aa20-48da-9eb3-0fb43f863049" containerName="main" containerID="cri-o://011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112" gracePeriod=60
Sep 16 17:57:25.289002 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.288978    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:57:25.393237 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393202    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-dshm\") pod \"a968e581-aa20-48da-9eb3-0fb43f863049\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") "
Sep 16 17:57:25.393402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393248    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-home\") pod \"a968e581-aa20-48da-9eb3-0fb43f863049\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") "
Sep 16 17:57:25.393402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393279    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a968e581-aa20-48da-9eb3-0fb43f863049-tls-certs\") pod \"a968e581-aa20-48da-9eb3-0fb43f863049\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") "
Sep 16 17:57:25.393402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393305    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmvv\" (UniqueName: \"kubernetes.io/projected/a968e581-aa20-48da-9eb3-0fb43f863049-kube-api-access-5tmvv\") pod \"a968e581-aa20-48da-9eb3-0fb43f863049\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") "
Sep 16 17:57:25.393402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393352    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-model-cache\") pod \"a968e581-aa20-48da-9eb3-0fb43f863049\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") "
Sep 16 17:57:25.393402 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393382    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-tmp-dir\") pod \"a968e581-aa20-48da-9eb3-0fb43f863049\" (UID: \"a968e581-aa20-48da-9eb3-0fb43f863049\") "
Sep 16 17:57:25.393680 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393547    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-home" (OuterVolumeSpecName: "home") pod "a968e581-aa20-48da-9eb3-0fb43f863049" (UID: "a968e581-aa20-48da-9eb3-0fb43f863049"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.393680 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-model-cache" (OuterVolumeSpecName: "model-cache") pod "a968e581-aa20-48da-9eb3-0fb43f863049" (UID: "a968e581-aa20-48da-9eb3-0fb43f863049"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.393793 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.393740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a968e581-aa20-48da-9eb3-0fb43f863049" (UID: "a968e581-aa20-48da-9eb3-0fb43f863049"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.395661 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.395632    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-dshm" (OuterVolumeSpecName: "dshm") pod "a968e581-aa20-48da-9eb3-0fb43f863049" (UID: "a968e581-aa20-48da-9eb3-0fb43f863049"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:57:25.395766 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.395659    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a968e581-aa20-48da-9eb3-0fb43f863049-kube-api-access-5tmvv" (OuterVolumeSpecName: "kube-api-access-5tmvv") pod "a968e581-aa20-48da-9eb3-0fb43f863049" (UID: "a968e581-aa20-48da-9eb3-0fb43f863049"). InnerVolumeSpecName "kube-api-access-5tmvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:57:25.395766 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.395750    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a968e581-aa20-48da-9eb3-0fb43f863049-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a968e581-aa20-48da-9eb3-0fb43f863049" (UID: "a968e581-aa20-48da-9eb3-0fb43f863049"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:57:25.494004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.493923    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.494004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.493947    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a968e581-aa20-48da-9eb3-0fb43f863049-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.494004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.493958    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmvv\" (UniqueName: \"kubernetes.io/projected/a968e581-aa20-48da-9eb3-0fb43f863049-kube-api-access-5tmvv\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.494004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.493967    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.494004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.493976    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.494004 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.493983    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a968e581-aa20-48da-9eb3-0fb43f863049-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:57:25.842466 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.842365    2560 generic.go:356] "Generic (PLEG): container finished" podID="a968e581-aa20-48da-9eb3-0fb43f863049" containerID="011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112" exitCode=0
Sep 16 17:57:25.842466 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.842418    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"
Sep 16 17:57:25.842734 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.842476    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" event={"ID":"a968e581-aa20-48da-9eb3-0fb43f863049","Type":"ContainerDied","Data":"011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"}
Sep 16 17:57:25.842734 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.842540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5" event={"ID":"a968e581-aa20-48da-9eb3-0fb43f863049","Type":"ContainerDied","Data":"ca04784ed2be6fa53330f388b4ed2e36768e4db56e8d0eb2e2528b936a625fa4"}
Sep 16 17:57:25.842734 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.842574    2560 scope.go:117] "RemoveContainer" containerID="011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"
Sep 16 17:57:25.854512 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.854489    2560 scope.go:117] "RemoveContainer" containerID="011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"
Sep 16 17:57:25.854862 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:57:25.854831    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112\": container with ID starting with 011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112 not found: ID does not exist" containerID="011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"
Sep 16 17:57:25.854963 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.854873    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112"} err="failed to get container status \"011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112\": rpc error: code = NotFound desc = could not find container \"011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112\": container with ID starting with 011f75c6ced6527c9c4ef083fb49a1cd43bb516cf8d630c17b7f5a201e136112 not found: ID does not exist"
Sep 16 17:57:25.897557 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.897526    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"]
Sep 16 17:57:25.908621 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:25.908581    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-prefill-8665687897-8bbk5"]
Sep 16 17:57:27.117336 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:27.117304    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a968e581-aa20-48da-9eb3-0fb43f863049" path="/var/lib/kubelet/pods/a968e581-aa20-48da-9eb3-0fb43f863049/volumes"
Sep 16 17:57:53.191228 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:57:53.191192    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"]
Sep 16 17:58:08.209572 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.209536    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc" podUID="719df1a8-82f6-4d3f-bca4-bfc0dea3319a" containerName="main" containerID="cri-o://4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d" gracePeriod=45
Sep 16 17:58:08.446000 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.445976    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:58:08.636429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636339    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-dshm\") pod \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") "
Sep 16 17:58:08.636429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636381    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wt7d\" (UniqueName: \"kubernetes.io/projected/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-kube-api-access-8wt7d\") pod \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") "
Sep 16 17:58:08.636429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tls-certs\") pod \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") "
Sep 16 17:58:08.636429 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-home\") pod \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") "
Sep 16 17:58:08.636795 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tmp-dir\") pod \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") "
Sep 16 17:58:08.636795 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-model-cache\") pod \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\" (UID: \"719df1a8-82f6-4d3f-bca4-bfc0dea3319a\") "
Sep 16 17:58:08.636964 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636840    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-home" (OuterVolumeSpecName: "home") pod "719df1a8-82f6-4d3f-bca4-bfc0dea3319a" (UID: "719df1a8-82f6-4d3f-bca4-bfc0dea3319a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.636964 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636857    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "719df1a8-82f6-4d3f-bca4-bfc0dea3319a" (UID: "719df1a8-82f6-4d3f-bca4-bfc0dea3319a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.636964 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.636948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-model-cache" (OuterVolumeSpecName: "model-cache") pod "719df1a8-82f6-4d3f-bca4-bfc0dea3319a" (UID: "719df1a8-82f6-4d3f-bca4-bfc0dea3319a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.638737 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.638711    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-kube-api-access-8wt7d" (OuterVolumeSpecName: "kube-api-access-8wt7d") pod "719df1a8-82f6-4d3f-bca4-bfc0dea3319a" (UID: "719df1a8-82f6-4d3f-bca4-bfc0dea3319a"). InnerVolumeSpecName "kube-api-access-8wt7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:58:08.638737 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.638714    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-dshm" (OuterVolumeSpecName: "dshm") pod "719df1a8-82f6-4d3f-bca4-bfc0dea3319a" (UID: "719df1a8-82f6-4d3f-bca4-bfc0dea3319a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:58:08.638926 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.638792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "719df1a8-82f6-4d3f-bca4-bfc0dea3319a" (UID: "719df1a8-82f6-4d3f-bca4-bfc0dea3319a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:58:08.737522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.737480    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.737522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.737516    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.737522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.737526    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wt7d\" (UniqueName: \"kubernetes.io/projected/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-kube-api-access-8wt7d\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.737522 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.737536    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.737795 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.737544    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:58:08.737795 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:08.737552    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719df1a8-82f6-4d3f-bca4-bfc0dea3319a-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:58:09.018776 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.018746    2560 generic.go:356] "Generic (PLEG): container finished" podID="719df1a8-82f6-4d3f-bca4-bfc0dea3319a" containerID="4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d" exitCode=0
Sep 16 17:58:09.018944 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.018790    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"
Sep 16 17:58:09.018944 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.018789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc" event={"ID":"719df1a8-82f6-4d3f-bca4-bfc0dea3319a","Type":"ContainerDied","Data":"4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d"}
Sep 16 17:58:09.019040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.018954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc" event={"ID":"719df1a8-82f6-4d3f-bca4-bfc0dea3319a","Type":"ContainerDied","Data":"4d54e14fcf8bf6d55829ed472196b17d1c86bded5aa6070a0a90032ab90c1b35"}
Sep 16 17:58:09.019040 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.018983    2560 scope.go:117] "RemoveContainer" containerID="4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d"
Sep 16 17:58:09.027913 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.027898    2560 scope.go:117] "RemoveContainer" containerID="4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d"
Sep 16 17:58:09.028164 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:58:09.028146    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d\": container with ID starting with 4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d not found: ID does not exist" containerID="4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d"
Sep 16 17:58:09.028254 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.028169    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d"} err="failed to get container status \"4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d\": rpc error: code = NotFound desc = could not find container \"4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d\": container with ID starting with 4d40a8aa5542e771c0c024b5c4b3c6ab04dfc6d4d49b913265437ba2533d6f8d not found: ID does not exist"
Sep 16 17:58:09.042067 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.042042    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"]
Sep 16 17:58:09.045575 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.045554    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-5hwwc"]
Sep 16 17:58:09.117127 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:09.117102    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="719df1a8-82f6-4d3f-bca4-bfc0dea3319a" path="/var/lib/kubelet/pods/719df1a8-82f6-4d3f-bca4-bfc0dea3319a/volumes"
Sep 16 17:58:46.430566 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430535    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"]
Sep 16 17:58:46.430962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430829    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a968e581-aa20-48da-9eb3-0fb43f863049" containerName="main"
Sep 16 17:58:46.430962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430841    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a968e581-aa20-48da-9eb3-0fb43f863049" containerName="main"
Sep 16 17:58:46.430962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430862    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="719df1a8-82f6-4d3f-bca4-bfc0dea3319a" containerName="main"
Sep 16 17:58:46.430962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430870    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="719df1a8-82f6-4d3f-bca4-bfc0dea3319a" containerName="main"
Sep 16 17:58:46.430962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430919    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="719df1a8-82f6-4d3f-bca4-bfc0dea3319a" containerName="main"
Sep 16 17:58:46.430962 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.430931    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a968e581-aa20-48da-9eb3-0fb43f863049" containerName="main"
Sep 16 17:58:46.434880 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.434860    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.438031 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.438012    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 17:58:46.438272 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.438253    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 17:58:46.438395 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.438374    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 17:58:46.438478 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.438387    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-s7h7d\""
Sep 16 17:58:46.481787 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.481722    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"]
Sep 16 17:58:46.518484 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.518455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p975z\" (UniqueName: \"kubernetes.io/projected/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-kube-api-access-p975z\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.518623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.518490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.518623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.518513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.518623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.518559    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.518623 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.518618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.518765 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.518657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.619532 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.619493    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p975z\" (UniqueName: \"kubernetes.io/projected/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-kube-api-access-p975z\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.619532 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.619534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.619786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.619655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.619786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.619698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.619786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.619725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.619786 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.619780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.620078 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.620055    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.620142 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.620086    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.620142 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.620107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.622068 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.622047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.622372 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.622354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.634316 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.634289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p975z\" (UniqueName: \"kubernetes.io/projected/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-kube-api-access-p975z\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.744509 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.744426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:46.872320 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.872286    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"]
Sep 16 17:58:46.875569 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.875554    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:46.885631 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.885611    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-x7zqj\""
Sep 16 17:58:46.909578 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.909557    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"]
Sep 16 17:58:46.922433 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.922406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:46.922531 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:46.922454    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqlw\" (UniqueName: \"kubernetes.io/projected/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-kube-api-access-rnqlw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:47.023646 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.023553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:47.023646 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.023618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqlw\" (UniqueName: \"kubernetes.io/projected/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-kube-api-access-rnqlw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:47.025921 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.025892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:47.053435 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.053414    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"]
Sep 16 17:58:47.054295 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:58:47.054270    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0ec8a3_d8da_4765_bef5_5c4e3dc50046.slice/crio-fb24d7d85801a56a2eebaaac01cef45c2e333231e2e5aba72f734e04f16731a3 WatchSource:0}: Error finding container fb24d7d85801a56a2eebaaac01cef45c2e333231e2e5aba72f734e04f16731a3: Status 404 returned error can't find the container with id fb24d7d85801a56a2eebaaac01cef45c2e333231e2e5aba72f734e04f16731a3
Sep 16 17:58:47.055758 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.055740    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqlw\" (UniqueName: \"kubernetes.io/projected/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-kube-api-access-rnqlw\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:47.166864 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.166827    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" event={"ID":"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046","Type":"ContainerStarted","Data":"3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"}
Sep 16 17:58:47.167028 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.166880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" event={"ID":"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046","Type":"ContainerStarted","Data":"fb24d7d85801a56a2eebaaac01cef45c2e333231e2e5aba72f734e04f16731a3"}
Sep 16 17:58:47.185325 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.185300    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:47.236438 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.236106    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" podStartSLOduration=1.23608709 podStartE2EDuration="1.23608709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:47.224510823 +0000 UTC m=+2042.633628071" watchObservedRunningTime="2026-09-16 17:58:47.23608709 +0000 UTC m=+2042.645204338"
Sep 16 17:58:47.521782 ip-10-0-128-156 kubenswrapper[2560]: W0916 17:58:47.521752    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bd04f6_865a_4c90_a8b1_7aa1bdd6421b.slice/crio-b39ecabe8557d7afdbc0972c2a77dbc0c8981bf0aa82a2e9a649232fedd14c00 WatchSource:0}: Error finding container b39ecabe8557d7afdbc0972c2a77dbc0c8981bf0aa82a2e9a649232fedd14c00: Status 404 returned error can't find the container with id b39ecabe8557d7afdbc0972c2a77dbc0c8981bf0aa82a2e9a649232fedd14c00
Sep 16 17:58:47.522177 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.521872    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"]
Sep 16 17:58:47.744928 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:47.744895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:48.172912 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:48.172872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" event={"ID":"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b","Type":"ContainerStarted","Data":"90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682"}
Sep 16 17:58:48.173087 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:48.172924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" event={"ID":"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b","Type":"ContainerStarted","Data":"b39ecabe8557d7afdbc0972c2a77dbc0c8981bf0aa82a2e9a649232fedd14c00"}
Sep 16 17:58:48.237010 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:48.236958    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" podStartSLOduration=2.2369420939999998 podStartE2EDuration="2.236942094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:58:48.235601757 +0000 UTC m=+2043.644719002" watchObservedRunningTime="2026-09-16 17:58:48.236942094 +0000 UTC m=+2043.646059341"
Sep 16 17:58:49.176761 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:49.176726    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:58:56.745140 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:56.745100    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:56.757278 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:56.757256    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:58:57.221058 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:58:57.221026    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:59:20.183176 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:20.183099    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 17:59:51.826648 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:51.826613    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"]
Sep 16 17:59:51.830507 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:51.830479    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"]
Sep 16 17:59:51.835913 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:59:51.835872    2560 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 17:59:51.835913 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:59:51.835913 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 17:59:51.835913 ip-10-0-128-156 kubenswrapper[2560]:  > containerID="3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1" cmd=["/bin/sleep","15"]
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:51.835928    2560 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" message="" err=<
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]:  >
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:59:51.835940    2560 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" podUID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" containerName="main" containerID="cri-o://3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"
Sep 16 17:59:51.836173 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:51.835986    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" podUID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" containerName="main" containerID="cri-o://3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1" gracePeriod=60
Sep 16 17:59:52.180682 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.180657    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:59:52.255112 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255085    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p975z\" (UniqueName: \"kubernetes.io/projected/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-kube-api-access-p975z\") pod \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") "
Sep 16 17:59:52.255112 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255123    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tmp-dir\") pod \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") "
Sep 16 17:59:52.255327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255179    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-home\") pod \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") "
Sep 16 17:59:52.255327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255194    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tls-certs\") pod \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") "
Sep 16 17:59:52.255327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-model-cache\") pod \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") "
Sep 16 17:59:52.255327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-dshm\") pod \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\" (UID: \"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046\") "
Sep 16 17:59:52.255550 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255449    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" (UID: "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.255550 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" (UID: "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.255686 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.255557    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-home" (OuterVolumeSpecName: "home") pod "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" (UID: "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.257257 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.257225    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-kube-api-access-p975z" (OuterVolumeSpecName: "kube-api-access-p975z") pod "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" (UID: "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046"). InnerVolumeSpecName "kube-api-access-p975z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:59:52.257371 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.257319    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-dshm" (OuterVolumeSpecName: "dshm") pod "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" (UID: "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:59:52.257413 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.257363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" (UID: "ac0ec8a3-d8da-4765-bef5-5c4e3dc50046"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:59:52.356574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.356513    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.356574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.356535    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.356574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.356545    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.356574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.356552    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.356574 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.356560    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p975z\" (UniqueName: \"kubernetes.io/projected/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-kube-api-access-p975z\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.356822 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.356618    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 17:59:52.436327 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.436295    2560 generic.go:356] "Generic (PLEG): container finished" podID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" containerID="3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1" exitCode=0
Sep 16 17:59:52.436465 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.436351    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"
Sep 16 17:59:52.436465 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.436387    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" event={"ID":"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046","Type":"ContainerDied","Data":"3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"}
Sep 16 17:59:52.436465 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.436440    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2" event={"ID":"ac0ec8a3-d8da-4765-bef5-5c4e3dc50046","Type":"ContainerDied","Data":"fb24d7d85801a56a2eebaaac01cef45c2e333231e2e5aba72f734e04f16731a3"}
Sep 16 17:59:52.436572 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.436468    2560 scope.go:117] "RemoveContainer" containerID="3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"
Sep 16 17:59:52.445445 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.445424    2560 scope.go:117] "RemoveContainer" containerID="3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"
Sep 16 17:59:52.445781 ip-10-0-128-156 kubenswrapper[2560]: E0916 17:59:52.445762    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1\": container with ID starting with 3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1 not found: ID does not exist" containerID="3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"
Sep 16 17:59:52.445843 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.445789    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1"} err="failed to get container status \"3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1\": rpc error: code = NotFound desc = could not find container \"3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1\": container with ID starting with 3daca480631f55357bb6599b8b09b00084b91e6e83ee90fa2481b4bd3679daf1 not found: ID does not exist"
Sep 16 17:59:52.484158 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.484122    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"]
Sep 16 17:59:52.495149 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:52.495122    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444phfm2"]
Sep 16 17:59:53.117732 ip-10-0-128-156 kubenswrapper[2560]: I0916 17:59:53.117700    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" path="/var/lib/kubelet/pods/ac0ec8a3-d8da-4765-bef5-5c4e3dc50046/volumes"
Sep 16 18:00:00.183571 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:00.183530    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:00:06.848553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:06.848497    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main" containerID="cri-o://90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682" gracePeriod=45
Sep 16 18:00:10.183937 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:10.183900    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:00:20.183006 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:20.182964    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:00:20.183452 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:20.183071    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 18:00:30.183315 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:30.183278    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:00:37.087740 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.087715    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 18:00:37.091080 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.091059    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnqlw\" (UniqueName: \"kubernetes.io/projected/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-kube-api-access-rnqlw\") pod \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") "
Sep 16 18:00:37.091195 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.091096    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-tls-certs\") pod \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\" (UID: \"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b\") "
Sep 16 18:00:37.093046 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.093019    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-kube-api-access-rnqlw" (OuterVolumeSpecName: "kube-api-access-rnqlw") pod "82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" (UID: "82bd04f6-865a-4c90-a8b1-7aa1bdd6421b"). InnerVolumeSpecName "kube-api-access-rnqlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:00:37.093175 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.093047    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" (UID: "82bd04f6-865a-4c90-a8b1-7aa1bdd6421b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:00:37.192181 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.192154    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqlw\" (UniqueName: \"kubernetes.io/projected/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-kube-api-access-rnqlw\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:00:37.192181 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.192176    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:00:37.617440 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.617403    2560 generic.go:356] "Generic (PLEG): container finished" podID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerID="90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682" exitCode=0
Sep 16 18:00:37.617631 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.617445    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"
Sep 16 18:00:37.617631 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.617487    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" event={"ID":"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b","Type":"ContainerDied","Data":"90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682"}
Sep 16 18:00:37.617631 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.617531    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh" event={"ID":"82bd04f6-865a-4c90-a8b1-7aa1bdd6421b","Type":"ContainerDied","Data":"b39ecabe8557d7afdbc0972c2a77dbc0c8981bf0aa82a2e9a649232fedd14c00"}
Sep 16 18:00:37.617631 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.617554    2560 scope.go:117] "RemoveContainer" containerID="90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682"
Sep 16 18:00:37.625807 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.625787    2560 scope.go:117] "RemoveContainer" containerID="90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682"
Sep 16 18:00:37.626082 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:00:37.626061    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682\": container with ID starting with 90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682 not found: ID does not exist" containerID="90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682"
Sep 16 18:00:37.626128 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.626090    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682"} err="failed to get container status \"90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682\": rpc error: code = NotFound desc = could not find container \"90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682\": container with ID starting with 90fb1e72bedebdee24893163a9d90f4f2948e0c8ad7ff3e808714b4fa1fa4682 not found: ID does not exist"
Sep 16 18:00:37.638833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.638807    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"]
Sep 16 18:00:37.642242 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:37.642220    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedx4fjh"]
Sep 16 18:00:39.117255 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:39.117221    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" path="/var/lib/kubelet/pods/82bd04f6-865a-4c90-a8b1-7aa1bdd6421b/volumes"
Sep 16 18:00:44.854307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854276    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"]
Sep 16 18:00:44.854689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854569    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" containerName="main"
Sep 16 18:00:44.854689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854579    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" containerName="main"
Sep 16 18:00:44.854689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854609    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main"
Sep 16 18:00:44.854689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854614    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main"
Sep 16 18:00:44.854689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854667    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac0ec8a3-d8da-4765-bef5-5c4e3dc50046" containerName="main"
Sep 16 18:00:44.854689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.854675    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82bd04f6-865a-4c90-a8b1-7aa1bdd6421b" containerName="main"
Sep 16 18:00:44.858800 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.858783    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:44.862460 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.862432    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 18:00:44.862928 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.862904    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 18:00:44.863019 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.862923    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-7l5jz\""
Sep 16 18:00:44.863019 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.862941    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 18:00:44.873953 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.873930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"]
Sep 16 18:00:44.940484 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.940448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e895ae12-f476-426d-99b9-7847186710de-tls-certs\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:44.940484 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.940485    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vw7\" (UniqueName: \"kubernetes.io/projected/e895ae12-f476-426d-99b9-7847186710de-kube-api-access-q9vw7\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:44.940722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.940517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-model-cache\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:44.940722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.940560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-home\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:44.940722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.940585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-tmp-dir\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:44.940722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:44.940643    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-dshm\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.040990 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.040943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e895ae12-f476-426d-99b9-7847186710de-tls-certs\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.040990 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.040988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vw7\" (UniqueName: \"kubernetes.io/projected/e895ae12-f476-426d-99b9-7847186710de-kube-api-access-q9vw7\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041238 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-model-cache\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041238 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-home\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041238 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-tmp-dir\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041238 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-dshm\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041454 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041432    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-home\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041524 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-model-cache\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.041573 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.041516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-tmp-dir\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.043191 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.043167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 18:00:45.043293 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.043242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-dshm\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.048309 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.048294    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 18:00:45.053828 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.053807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e895ae12-f476-426d-99b9-7847186710de-tls-certs\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.059979 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.059964    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 18:00:45.069744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.069725    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vw7\" (UniqueName: \"kubernetes.io/projected/e895ae12-f476-426d-99b9-7847186710de-kube-api-access-q9vw7\") pod \"decom-v2-kserve-77f867d787-gccgg\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.171390 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.171309    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-7l5jz\""
Sep 16 18:00:45.179062 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.179038    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:45.473605 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.473555    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"]
Sep 16 18:00:45.476526 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:00:45.476496    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode895ae12_f476_426d_99b9_7847186710de.slice/crio-86deb5d6e7f214c428285b1faa3b0b6ec3208519e592aa24993204b70a0066d3 WatchSource:0}: Error finding container 86deb5d6e7f214c428285b1faa3b0b6ec3208519e592aa24993204b70a0066d3: Status 404 returned error can't find the container with id 86deb5d6e7f214c428285b1faa3b0b6ec3208519e592aa24993204b70a0066d3
Sep 16 18:00:45.650788 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.650751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg" event={"ID":"e895ae12-f476-426d-99b9-7847186710de","Type":"ContainerStarted","Data":"00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c"}
Sep 16 18:00:45.650933 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.650807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg" event={"ID":"e895ae12-f476-426d-99b9-7847186710de","Type":"ContainerStarted","Data":"86deb5d6e7f214c428285b1faa3b0b6ec3208519e592aa24993204b70a0066d3"}
Sep 16 18:00:45.669209 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:45.669152    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg" podStartSLOduration=1.669137015 podStartE2EDuration="1.669137015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:00:45.667670988 +0000 UTC m=+2161.076788236" watchObservedRunningTime="2026-09-16 18:00:45.669137015 +0000 UTC m=+2161.078254263"
Sep 16 18:00:46.179388 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:46.179338    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:55.179880 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:55.179849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:55.192568 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:55.192545    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:55.700858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:55.700828    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:00:59.661051 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.661017    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"]
Sep 16 18:00:59.664106 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.664089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.666978 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.666957    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 16 18:00:59.667125 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.667018    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 18:00:59.667125 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.667086    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:00:59.667310 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.667293    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 18:00:59.667713 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.667689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 18:00:59.667832 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.667817    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-8xfng\""
Sep 16 18:00:59.687164 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.687136    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"]
Sep 16 18:00:59.758294 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fdb58e7-4373-47fe-b43a-699137a926ea-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758457 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758302    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-home\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758457 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-model-cache\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758457 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758400    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rtxg\" (UniqueName: \"kubernetes.io/projected/1fdb58e7-4373-47fe-b43a-699137a926ea-kube-api-access-2rtxg\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758457 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdb58e7-4373-47fe-b43a-699137a926ea-tls-certs\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758618 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758618 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.758618 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.758525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-dshm\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859674 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859827 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-dshm\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859827 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fdb58e7-4373-47fe-b43a-699137a926ea-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859827 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859747    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-home\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859827 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859765    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-model-cache\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859827 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859795    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rtxg\" (UniqueName: \"kubernetes.io/projected/1fdb58e7-4373-47fe-b43a-699137a926ea-kube-api-access-2rtxg\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.859827 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdb58e7-4373-47fe-b43a-699137a926ea-tls-certs\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.860189 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.859860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.860189 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.860115    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.860189 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.860137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-home\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.860423 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.860405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.860508 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.860405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-model-cache\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.860683 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.860661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fdb58e7-4373-47fe-b43a-699137a926ea-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.861990 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.861972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-dshm\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.862338 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.862317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdb58e7-4373-47fe-b43a-699137a926ea-tls-certs\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.868497 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.868473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rtxg\" (UniqueName: \"kubernetes.io/projected/1fdb58e7-4373-47fe-b43a-699137a926ea-kube-api-access-2rtxg\") pod \"tokenizer-clean-path-test-kserve-d975798d6-44zff\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:00:59.974402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:00:59.974371    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:01:00.260749 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:00.260721    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"]
Sep 16 18:01:00.263128 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:01:00.263093    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdb58e7_4373_47fe_b43a_699137a926ea.slice/crio-32c5e4b18ee676214e4dba7e2900a1402ad7f0bd30018be508ec5244b673dc6e WatchSource:0}: Error finding container 32c5e4b18ee676214e4dba7e2900a1402ad7f0bd30018be508ec5244b673dc6e: Status 404 returned error can't find the container with id 32c5e4b18ee676214e4dba7e2900a1402ad7f0bd30018be508ec5244b673dc6e
Sep 16 18:01:00.710658 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:00.710626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" event={"ID":"1fdb58e7-4373-47fe-b43a-699137a926ea","Type":"ContainerStarted","Data":"e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a"}
Sep 16 18:01:00.711033 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:00.710678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" event={"ID":"1fdb58e7-4373-47fe-b43a-699137a926ea","Type":"ContainerStarted","Data":"32c5e4b18ee676214e4dba7e2900a1402ad7f0bd30018be508ec5244b673dc6e"}
Sep 16 18:01:02.719863 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:02.719828    2560 generic.go:356] "Generic (PLEG): container finished" podID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerID="e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a" exitCode=0
Sep 16 18:01:02.720272 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:02.719906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" event={"ID":"1fdb58e7-4373-47fe-b43a-699137a926ea","Type":"ContainerDied","Data":"e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a"}
Sep 16 18:01:03.725569 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:03.725535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" event={"ID":"1fdb58e7-4373-47fe-b43a-699137a926ea","Type":"ContainerStarted","Data":"7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"}
Sep 16 18:01:03.745820 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:03.745763    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" podStartSLOduration=2.508478645 podStartE2EDuration="4.745744704s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.237266059s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:01:03.745023012 +0000 UTC m=+2179.154140259" watchObservedRunningTime="2026-09-16 18:01:03.745744704 +0000 UTC m=+2179.154861953"
Sep 16 18:01:03.974526 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:03.974483    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:01:09.974809 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:09.974778    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:01:09.986945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:09.986920    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:01:10.766398 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:10.766368    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:01:56.896295 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:01:56.896251    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"]
Sep 16 18:02:11.914705 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:11.914582    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg" podUID="e895ae12-f476-426d-99b9-7847186710de" containerName="main" containerID="cri-o://00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c" gracePeriod=45
Sep 16 18:02:12.166984 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.166908    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:02:12.240228 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-model-cache\") pod \"e895ae12-f476-426d-99b9-7847186710de\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") "
Sep 16 18:02:12.240402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240240    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-dshm\") pod \"e895ae12-f476-426d-99b9-7847186710de\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") "
Sep 16 18:02:12.240402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240272    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9vw7\" (UniqueName: \"kubernetes.io/projected/e895ae12-f476-426d-99b9-7847186710de-kube-api-access-q9vw7\") pod \"e895ae12-f476-426d-99b9-7847186710de\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") "
Sep 16 18:02:12.240402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-home\") pod \"e895ae12-f476-426d-99b9-7847186710de\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") "
Sep 16 18:02:12.240402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240334    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e895ae12-f476-426d-99b9-7847186710de-tls-certs\") pod \"e895ae12-f476-426d-99b9-7847186710de\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") "
Sep 16 18:02:12.240402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240387    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-tmp-dir\") pod \"e895ae12-f476-426d-99b9-7847186710de\" (UID: \"e895ae12-f476-426d-99b9-7847186710de\") "
Sep 16 18:02:12.240717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240434    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-model-cache" (OuterVolumeSpecName: "model-cache") pod "e895ae12-f476-426d-99b9-7847186710de" (UID: "e895ae12-f476-426d-99b9-7847186710de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:12.240717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240567    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-home" (OuterVolumeSpecName: "home") pod "e895ae12-f476-426d-99b9-7847186710de" (UID: "e895ae12-f476-426d-99b9-7847186710de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:12.240717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240678    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:02:12.240717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240697    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:02:12.240931 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.240780    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e895ae12-f476-426d-99b9-7847186710de" (UID: "e895ae12-f476-426d-99b9-7847186710de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:12.242667 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.242631    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-dshm" (OuterVolumeSpecName: "dshm") pod "e895ae12-f476-426d-99b9-7847186710de" (UID: "e895ae12-f476-426d-99b9-7847186710de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:02:12.242797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.242731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e895ae12-f476-426d-99b9-7847186710de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e895ae12-f476-426d-99b9-7847186710de" (UID: "e895ae12-f476-426d-99b9-7847186710de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:02:12.242797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.242742    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e895ae12-f476-426d-99b9-7847186710de-kube-api-access-q9vw7" (OuterVolumeSpecName: "kube-api-access-q9vw7") pod "e895ae12-f476-426d-99b9-7847186710de" (UID: "e895ae12-f476-426d-99b9-7847186710de"). InnerVolumeSpecName "kube-api-access-q9vw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:02:12.341730 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.341694    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:02:12.341730 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.341724    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vw7\" (UniqueName: \"kubernetes.io/projected/e895ae12-f476-426d-99b9-7847186710de-kube-api-access-q9vw7\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:02:12.341730 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.341734    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e895ae12-f476-426d-99b9-7847186710de-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:02:12.341970 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:12.341744    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e895ae12-f476-426d-99b9-7847186710de-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:02:13.006405 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.006367    2560 generic.go:356] "Generic (PLEG): container finished" podID="e895ae12-f476-426d-99b9-7847186710de" containerID="00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c" exitCode=0
Sep 16 18:02:13.006405 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.006410    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"
Sep 16 18:02:13.006989 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.006445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg" event={"ID":"e895ae12-f476-426d-99b9-7847186710de","Type":"ContainerDied","Data":"00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c"}
Sep 16 18:02:13.006989 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.006495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg" event={"ID":"e895ae12-f476-426d-99b9-7847186710de","Type":"ContainerDied","Data":"86deb5d6e7f214c428285b1faa3b0b6ec3208519e592aa24993204b70a0066d3"}
Sep 16 18:02:13.006989 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.006519    2560 scope.go:117] "RemoveContainer" containerID="00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c"
Sep 16 18:02:13.015100 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.015083    2560 scope.go:117] "RemoveContainer" containerID="00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c"
Sep 16 18:02:13.015371 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:02:13.015348    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c\": container with ID starting with 00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c not found: ID does not exist" containerID="00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c"
Sep 16 18:02:13.015432 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.015382    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c"} err="failed to get container status \"00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c\": rpc error: code = NotFound desc = could not find container \"00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c\": container with ID starting with 00f08bc33e925ce14fd3544a09ff947a6f053f0579a7fd7e0b12ed085f8d8f4c not found: ID does not exist"
Sep 16 18:02:13.027789 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.027765    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"]
Sep 16 18:02:13.031677 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.031657    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-gccgg"]
Sep 16 18:02:13.123125 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:13.123076    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e895ae12-f476-426d-99b9-7847186710de" path="/var/lib/kubelet/pods/e895ae12-f476-426d-99b9-7847186710de/volumes"
Sep 16 18:02:17.165208 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.165172    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"]
Sep 16 18:02:17.165695 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.165642    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e895ae12-f476-426d-99b9-7847186710de" containerName="main"
Sep 16 18:02:17.165695 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.165661    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e895ae12-f476-426d-99b9-7847186710de" containerName="main"
Sep 16 18:02:17.165820 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.165739    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e895ae12-f476-426d-99b9-7847186710de" containerName="main"
Sep 16 18:02:17.168457 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.168435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.171618 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.171576    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 18:02:17.171715 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.171624    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Sep 16 18:02:17.171715 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.171702    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 18:02:17.171795 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.171758    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-rfbss\""
Sep 16 18:02:17.192485 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.192465    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"]
Sep 16 18:02:17.283574 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.283546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gg9\" (UniqueName: \"kubernetes.io/projected/e663e6df-0d61-4412-a827-c8b9e883e20c-kube-api-access-w9gg9\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.283722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.283604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-dshm\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.283722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.283641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-home\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.283722 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.283657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.283832 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.283725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e663e6df-0d61-4412-a827-c8b9e883e20c-tls-certs\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.283832 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.283761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-model-cache\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384231 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e663e6df-0d61-4412-a827-c8b9e883e20c-tls-certs\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384377 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-model-cache\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384377 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gg9\" (UniqueName: \"kubernetes.io/projected/e663e6df-0d61-4412-a827-c8b9e883e20c-kube-api-access-w9gg9\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384377 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-dshm\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384563 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384405    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-home\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384563 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384774 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384750    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-model-cache\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384835 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.384891 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.384826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-home\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.386536 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.386517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-dshm\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.386724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.386709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e663e6df-0d61-4412-a827-c8b9e883e20c-tls-certs\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.393257 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.393235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gg9\" (UniqueName: \"kubernetes.io/projected/e663e6df-0d61-4412-a827-c8b9e883e20c-kube-api-access-w9gg9\") pod \"leave-v1-kserve-65684d4f5-trlld\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.478293 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.478268    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:17.761265 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:17.761240    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"]
Sep 16 18:02:17.764765 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:02:17.764734    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode663e6df_0d61_4412_a827_c8b9e883e20c.slice/crio-36433d38b33beba3803d89c90705e64406191ff2f5e2226774feddf4a13c8110 WatchSource:0}: Error finding container 36433d38b33beba3803d89c90705e64406191ff2f5e2226774feddf4a13c8110: Status 404 returned error can't find the container with id 36433d38b33beba3803d89c90705e64406191ff2f5e2226774feddf4a13c8110
Sep 16 18:02:18.030406 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:18.030328    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" event={"ID":"e663e6df-0d61-4412-a827-c8b9e883e20c","Type":"ContainerStarted","Data":"9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85"}
Sep 16 18:02:18.030406 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:18.030379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" event={"ID":"e663e6df-0d61-4412-a827-c8b9e883e20c","Type":"ContainerStarted","Data":"36433d38b33beba3803d89c90705e64406191ff2f5e2226774feddf4a13c8110"}
Sep 16 18:02:18.052121 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:18.052057    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" podStartSLOduration=1.052036091 podStartE2EDuration="1.052036091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:02:18.049835044 +0000 UTC m=+2253.458952293" watchObservedRunningTime="2026-09-16 18:02:18.052036091 +0000 UTC m=+2253.461153340"
Sep 16 18:02:18.478674 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:18.478641    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:27.478727 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:27.478688    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:27.490886 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:27.490859    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:28.081819 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:28.081789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:02:32.950524 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:32.950485    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"]
Sep 16 18:02:32.953000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:32.952980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:32.954858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:32.954834    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 18:02:32.964562 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:32.964538    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"]
Sep 16 18:02:33.007724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.007688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-dshm\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.007724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.007725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4f43c-0281-4152-9082-6460ccbfe0fa-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.007938 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.007754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-model-cache\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.007938 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.007772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.007938 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.007828    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-home\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.007938 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.007844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4dl\" (UniqueName: \"kubernetes.io/projected/33b4f43c-0281-4152-9082-6460ccbfe0fa-kube-api-access-hk4dl\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-home\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4dl\" (UniqueName: \"kubernetes.io/projected/33b4f43c-0281-4152-9082-6460ccbfe0fa-kube-api-access-hk4dl\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-dshm\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4f43c-0281-4152-9082-6460ccbfe0fa-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-model-cache\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109574 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-home\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109672 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-model-cache\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.109737 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.109692    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.111447 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.111421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-dshm\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.111650 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.111633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4f43c-0281-4152-9082-6460ccbfe0fa-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.118222 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.118190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4dl\" (UniqueName: \"kubernetes.io/projected/33b4f43c-0281-4152-9082-6460ccbfe0fa-kube-api-access-hk4dl\") pod \"leave-v2-kserve-556b4dddf9-wn6x6\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.264478 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.264440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:33.534612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:33.534510    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"]
Sep 16 18:02:33.538179 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:02:33.538145    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b4f43c_0281_4152_9082_6460ccbfe0fa.slice/crio-a7e1bf2c45260cd5731da00db11b71897777190160ec58f8cb139920f1761e49 WatchSource:0}: Error finding container a7e1bf2c45260cd5731da00db11b71897777190160ec58f8cb139920f1761e49: Status 404 returned error can't find the container with id a7e1bf2c45260cd5731da00db11b71897777190160ec58f8cb139920f1761e49
Sep 16 18:02:34.096826 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:34.096785    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6" event={"ID":"33b4f43c-0281-4152-9082-6460ccbfe0fa","Type":"ContainerStarted","Data":"d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd"}
Sep 16 18:02:34.097226 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:34.096847    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6" event={"ID":"33b4f43c-0281-4152-9082-6460ccbfe0fa","Type":"ContainerStarted","Data":"a7e1bf2c45260cd5731da00db11b71897777190160ec58f8cb139920f1761e49"}
Sep 16 18:02:34.117497 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:34.117449    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6" podStartSLOduration=2.117432682 podStartE2EDuration="2.117432682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:02:34.115307047 +0000 UTC m=+2269.524424296" watchObservedRunningTime="2026-09-16 18:02:34.117432682 +0000 UTC m=+2269.526549929"
Sep 16 18:02:34.265239 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:34.265201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:43.265039 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:43.265001    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:43.277419 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:43.277391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:02:44.150151 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:02:44.150116    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:03:02.231485 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.231441    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"]
Sep 16 18:03:02.243276 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:03:02.243229    2560 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:03:02.243276 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:03:02.243276 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:03:02.243276 ip-10-0-128-156 kubenswrapper[2560]:  > containerID="7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac" cmd=["/bin/sleep","15"]
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.243288    2560 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" message="" err=<
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]:  >
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:03:02.243301    2560 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="main" containerID="cri-o://7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"
Sep 16 18:03:02.243604 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.243330    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="main" containerID="cri-o://7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac" gracePeriod=60
Sep 16 18:03:02.589775 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.589752    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:03:02.660234 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660207    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-model-cache\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660360 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660264    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdb58e7-4373-47fe-b43a-699137a926ea-tls-certs\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660360 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-kserve-provision-location\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660360 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660307    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-dshm\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660360 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660356    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fdb58e7-4373-47fe-b43a-699137a926ea-cabundle-cert\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660390    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-tmp-dir\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660415    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-home\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660435    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rtxg\" (UniqueName: \"kubernetes.io/projected/1fdb58e7-4373-47fe-b43a-699137a926ea-kube-api-access-2rtxg\") pod \"1fdb58e7-4373-47fe-b43a-699137a926ea\" (UID: \"1fdb58e7-4373-47fe-b43a-699137a926ea\") "
Sep 16 18:03:02.660567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.660816 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660700    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.660816 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660737    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdb58e7-4373-47fe-b43a-699137a926ea-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:03:02.660816 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660749    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-home" (OuterVolumeSpecName: "home") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.660816 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.660739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.662503 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.662477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fdb58e7-4373-47fe-b43a-699137a926ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:03:02.662658 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.662539    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-dshm" (OuterVolumeSpecName: "dshm") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.662658 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.662551    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdb58e7-4373-47fe-b43a-699137a926ea-kube-api-access-2rtxg" (OuterVolumeSpecName: "kube-api-access-2rtxg") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "kube-api-access-2rtxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:03:02.679765 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.679739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fdb58e7-4373-47fe-b43a-699137a926ea" (UID: "1fdb58e7-4373-47fe-b43a-699137a926ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:02.761901 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761851    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fdb58e7-4373-47fe-b43a-699137a926ea-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.761901 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761872    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.761901 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761881    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.761901 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761890    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rtxg\" (UniqueName: \"kubernetes.io/projected/1fdb58e7-4373-47fe-b43a-699137a926ea-kube-api-access-2rtxg\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.761901 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761899    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fdb58e7-4373-47fe-b43a-699137a926ea-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.762116 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761908    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:02.762116 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:02.761916    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fdb58e7-4373-47fe-b43a-699137a926ea-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:03.219221 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.219178    2560 generic.go:356] "Generic (PLEG): container finished" podID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerID="7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac" exitCode=0
Sep 16 18:03:03.219386 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.219236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"
Sep 16 18:03:03.219386 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.219259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" event={"ID":"1fdb58e7-4373-47fe-b43a-699137a926ea","Type":"ContainerDied","Data":"7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"}
Sep 16 18:03:03.219386 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.219329    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff" event={"ID":"1fdb58e7-4373-47fe-b43a-699137a926ea","Type":"ContainerDied","Data":"32c5e4b18ee676214e4dba7e2900a1402ad7f0bd30018be508ec5244b673dc6e"}
Sep 16 18:03:03.219386 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.219362    2560 scope.go:117] "RemoveContainer" containerID="7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"
Sep 16 18:03:03.229035 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.229012    2560 scope.go:117] "RemoveContainer" containerID="e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a"
Sep 16 18:03:03.238495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.238477    2560 scope.go:117] "RemoveContainer" containerID="7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"
Sep 16 18:03:03.238865 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:03:03.238841    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac\": container with ID starting with 7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac not found: ID does not exist" containerID="7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"
Sep 16 18:03:03.238922 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.238875    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac"} err="failed to get container status \"7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac\": rpc error: code = NotFound desc = could not find container \"7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac\": container with ID starting with 7cb61b6150c6f2943fa76702e2c6c200c79885fa011c9053a676219966076fac not found: ID does not exist"
Sep 16 18:03:03.238922 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.238901    2560 scope.go:117] "RemoveContainer" containerID="e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a"
Sep 16 18:03:03.239232 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:03:03.239197    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a\": container with ID starting with e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a not found: ID does not exist" containerID="e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a"
Sep 16 18:03:03.239292 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.239242    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a"} err="failed to get container status \"e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a\": rpc error: code = NotFound desc = could not find container \"e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a\": container with ID starting with e25008845ef000936d102f438cc33d31a993acc8d67fb314d947b8d42452f54a not found: ID does not exist"
Sep 16 18:03:03.278875 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.278854    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"]
Sep 16 18:03:03.314051 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:03.314026    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-d975798d6-44zff"]
Sep 16 18:03:05.117656 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:05.117621    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" path="/var/lib/kubelet/pods/1fdb58e7-4373-47fe-b43a-699137a926ea/volumes"
Sep 16 18:03:06.948472 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:06.948437    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"]
Sep 16 18:03:10.682016 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:10.681982    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"]
Sep 16 18:03:12.732056 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.732023    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"]
Sep 16 18:03:12.732418 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.732326    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="storage-initializer"
Sep 16 18:03:12.732418 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.732338    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="storage-initializer"
Sep 16 18:03:12.732418 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.732358    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="main"
Sep 16 18:03:12.732418 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.732364    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="main"
Sep 16 18:03:12.732418 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.732420    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fdb58e7-4373-47fe-b43a-699137a926ea" containerName="main"
Sep 16 18:03:12.735406 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.735389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.739252 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.739228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Sep 16 18:03:12.739798 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.739780    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Sep 16 18:03:12.739858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.739796    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v1-kserve-self-signed-certs\""
Sep 16 18:03:12.739858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.739847    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-ll8h8\""
Sep 16 18:03:12.764021 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.763997    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"]
Sep 16 18:03:12.844312 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.844285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.844411 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.844320    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-home\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.844411 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.844371    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.844486 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.844442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-dshm\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.844486 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.844462    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46d4737-9c08-423f-88b6-d5ee457bba4b-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.844486 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.844478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwz7\" (UniqueName: \"kubernetes.io/projected/c46d4737-9c08-423f-88b6-d5ee457bba4b-kube-api-access-htwz7\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945143 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945118    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945282 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945159    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-dshm\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945282 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46d4737-9c08-423f-88b6-d5ee457bba4b-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwz7\" (UniqueName: \"kubernetes.io/projected/c46d4737-9c08-423f-88b6-d5ee457bba4b-kube-api-access-htwz7\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-home\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945559 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945479    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-model-cache\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945734 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-home\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.945786 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.945732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-tmp-dir\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.947541 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.947519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-dshm\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.947741 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.947725    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46d4737-9c08-423f-88b6-d5ee457bba4b-tls-certs\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:12.959448 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:12.959429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwz7\" (UniqueName: \"kubernetes.io/projected/c46d4737-9c08-423f-88b6-d5ee457bba4b-kube-api-access-htwz7\") pod \"tri-v1-kserve-79f8f98bc5-w9k4p\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") " pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:13.045360 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:13.045299    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:13.337286 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:03:13.337253    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46d4737_9c08_423f_88b6_d5ee457bba4b.slice/crio-7ca65cde88079a79475dc6a58b28b7267216e0ee379db9ce0221e41b258c5334 WatchSource:0}: Error finding container 7ca65cde88079a79475dc6a58b28b7267216e0ee379db9ce0221e41b258c5334: Status 404 returned error can't find the container with id 7ca65cde88079a79475dc6a58b28b7267216e0ee379db9ce0221e41b258c5334
Sep 16 18:03:13.338094 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:13.338067    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"]
Sep 16 18:03:14.265301 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:14.265270    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p" event={"ID":"c46d4737-9c08-423f-88b6-d5ee457bba4b","Type":"ContainerStarted","Data":"bca67ecc018a77694955789869df925c76bd8a5ff6b9553d7e4102360d7de09a"}
Sep 16 18:03:14.265671 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:14.265323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p" event={"ID":"c46d4737-9c08-423f-88b6-d5ee457bba4b","Type":"ContainerStarted","Data":"7ca65cde88079a79475dc6a58b28b7267216e0ee379db9ce0221e41b258c5334"}
Sep 16 18:03:14.301790 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:14.301742    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p" podStartSLOduration=2.3017270180000002 podStartE2EDuration="2.301727018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:03:14.300235829 +0000 UTC m=+2309.709353055" watchObservedRunningTime="2026-09-16 18:03:14.301727018 +0000 UTC m=+2309.710844269"
Sep 16 18:03:15.045684 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:15.045648    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:21.967012 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:21.966944    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" podUID="e663e6df-0d61-4412-a827-c8b9e883e20c" containerName="main" containerID="cri-o://9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85" gracePeriod=45
Sep 16 18:03:22.070973 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.070937    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" podUID="e663e6df-0d61-4412-a827-c8b9e883e20c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Sep 16 18:03:22.212054 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.212026    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:03:22.297332 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.297243    2560 generic.go:356] "Generic (PLEG): container finished" podID="e663e6df-0d61-4412-a827-c8b9e883e20c" containerID="9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85" exitCode=0
Sep 16 18:03:22.297332 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.297290    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"
Sep 16 18:03:22.297493 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.297326    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" event={"ID":"e663e6df-0d61-4412-a827-c8b9e883e20c","Type":"ContainerDied","Data":"9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85"}
Sep 16 18:03:22.297493 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.297379    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld" event={"ID":"e663e6df-0d61-4412-a827-c8b9e883e20c","Type":"ContainerDied","Data":"36433d38b33beba3803d89c90705e64406191ff2f5e2226774feddf4a13c8110"}
Sep 16 18:03:22.297493 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.297401    2560 scope.go:117] "RemoveContainer" containerID="9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85"
Sep 16 18:03:22.305120 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.305103    2560 scope.go:117] "RemoveContainer" containerID="9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85"
Sep 16 18:03:22.305391 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:03:22.305374    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85\": container with ID starting with 9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85 not found: ID does not exist" containerID="9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85"
Sep 16 18:03:22.305436 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.305401    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85"} err="failed to get container status \"9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85\": rpc error: code = NotFound desc = could not find container \"9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85\": container with ID starting with 9372e93c09653106e0b8a364916ab6bacebae86761cc07f94aa3fc2d96e4ab85 not found: ID does not exist"
Sep 16 18:03:22.322697 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e663e6df-0d61-4412-a827-c8b9e883e20c-tls-certs\") pod \"e663e6df-0d61-4412-a827-c8b9e883e20c\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") "
Sep 16 18:03:22.322788 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322703    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-model-cache\") pod \"e663e6df-0d61-4412-a827-c8b9e883e20c\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") "
Sep 16 18:03:22.322788 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322723    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9gg9\" (UniqueName: \"kubernetes.io/projected/e663e6df-0d61-4412-a827-c8b9e883e20c-kube-api-access-w9gg9\") pod \"e663e6df-0d61-4412-a827-c8b9e883e20c\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") "
Sep 16 18:03:22.322788 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322764    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-home\") pod \"e663e6df-0d61-4412-a827-c8b9e883e20c\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") "
Sep 16 18:03:22.322940 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-dshm\") pod \"e663e6df-0d61-4412-a827-c8b9e883e20c\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") "
Sep 16 18:03:22.322940 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322853    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-tmp-dir\") pod \"e663e6df-0d61-4412-a827-c8b9e883e20c\" (UID: \"e663e6df-0d61-4412-a827-c8b9e883e20c\") "
Sep 16 18:03:22.323054 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.322934    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e663e6df-0d61-4412-a827-c8b9e883e20c" (UID: "e663e6df-0d61-4412-a827-c8b9e883e20c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:22.323121 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.323084    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:22.323180 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.323112    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-home" (OuterVolumeSpecName: "home") pod "e663e6df-0d61-4412-a827-c8b9e883e20c" (UID: "e663e6df-0d61-4412-a827-c8b9e883e20c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:22.323180 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.323126    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e663e6df-0d61-4412-a827-c8b9e883e20c" (UID: "e663e6df-0d61-4412-a827-c8b9e883e20c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:22.324773 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.324752    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e663e6df-0d61-4412-a827-c8b9e883e20c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e663e6df-0d61-4412-a827-c8b9e883e20c" (UID: "e663e6df-0d61-4412-a827-c8b9e883e20c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:03:22.324918 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.324900    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e663e6df-0d61-4412-a827-c8b9e883e20c-kube-api-access-w9gg9" (OuterVolumeSpecName: "kube-api-access-w9gg9") pod "e663e6df-0d61-4412-a827-c8b9e883e20c" (UID: "e663e6df-0d61-4412-a827-c8b9e883e20c"). InnerVolumeSpecName "kube-api-access-w9gg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:03:22.324998 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.324975    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-dshm" (OuterVolumeSpecName: "dshm") pod "e663e6df-0d61-4412-a827-c8b9e883e20c" (UID: "e663e6df-0d61-4412-a827-c8b9e883e20c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:22.424352 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.424327    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:22.424443 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.424367    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:22.424443 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.424377    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e663e6df-0d61-4412-a827-c8b9e883e20c-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:22.424443 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.424387    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9gg9\" (UniqueName: \"kubernetes.io/projected/e663e6df-0d61-4412-a827-c8b9e883e20c-kube-api-access-w9gg9\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:22.424443 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.424396    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e663e6df-0d61-4412-a827-c8b9e883e20c-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:22.623992 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.623962    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"]
Sep 16 18:03:22.634452 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:22.634429    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-trlld"]
Sep 16 18:03:23.045735 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:23.045703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:23.057841 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:23.057813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:23.117227 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:23.117201    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e663e6df-0d61-4412-a827-c8b9e883e20c" path="/var/lib/kubelet/pods/e663e6df-0d61-4412-a827-c8b9e883e20c/volumes"
Sep 16 18:03:23.313250 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:23.313163    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:03:25.700384 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:25.700343    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6" podUID="33b4f43c-0281-4152-9082-6460ccbfe0fa" containerName="main" containerID="cri-o://d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd" gracePeriod=45
Sep 16 18:03:25.938203 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:25.938173    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:03:26.053520 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053485    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-model-cache\") pod \"33b4f43c-0281-4152-9082-6460ccbfe0fa\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") "
Sep 16 18:03:26.053520 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053524    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4f43c-0281-4152-9082-6460ccbfe0fa-tls-certs\") pod \"33b4f43c-0281-4152-9082-6460ccbfe0fa\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") "
Sep 16 18:03:26.053809 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053556    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-tmp-dir\") pod \"33b4f43c-0281-4152-9082-6460ccbfe0fa\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") "
Sep 16 18:03:26.053809 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-home\") pod \"33b4f43c-0281-4152-9082-6460ccbfe0fa\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") "
Sep 16 18:03:26.053809 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-dshm\") pod \"33b4f43c-0281-4152-9082-6460ccbfe0fa\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") "
Sep 16 18:03:26.053809 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053702    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk4dl\" (UniqueName: \"kubernetes.io/projected/33b4f43c-0281-4152-9082-6460ccbfe0fa-kube-api-access-hk4dl\") pod \"33b4f43c-0281-4152-9082-6460ccbfe0fa\" (UID: \"33b4f43c-0281-4152-9082-6460ccbfe0fa\") "
Sep 16 18:03:26.053996 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "33b4f43c-0281-4152-9082-6460ccbfe0fa" (UID: "33b4f43c-0281-4152-9082-6460ccbfe0fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:26.053996 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.053932    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33b4f43c-0281-4152-9082-6460ccbfe0fa" (UID: "33b4f43c-0281-4152-9082-6460ccbfe0fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:26.054078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.054010    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-home" (OuterVolumeSpecName: "home") pod "33b4f43c-0281-4152-9082-6460ccbfe0fa" (UID: "33b4f43c-0281-4152-9082-6460ccbfe0fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:26.055785 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.055757    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33b4f43c-0281-4152-9082-6460ccbfe0fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33b4f43c-0281-4152-9082-6460ccbfe0fa" (UID: "33b4f43c-0281-4152-9082-6460ccbfe0fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:03:26.055785 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.055764    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-dshm" (OuterVolumeSpecName: "dshm") pod "33b4f43c-0281-4152-9082-6460ccbfe0fa" (UID: "33b4f43c-0281-4152-9082-6460ccbfe0fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:03:26.055942 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.055854    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b4f43c-0281-4152-9082-6460ccbfe0fa-kube-api-access-hk4dl" (OuterVolumeSpecName: "kube-api-access-hk4dl") pod "33b4f43c-0281-4152-9082-6460ccbfe0fa" (UID: "33b4f43c-0281-4152-9082-6460ccbfe0fa"). InnerVolumeSpecName "kube-api-access-hk4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:03:26.154557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.154501    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:26.154557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.154550    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:26.154557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.154562    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:26.154825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.154575    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk4dl\" (UniqueName: \"kubernetes.io/projected/33b4f43c-0281-4152-9082-6460ccbfe0fa-kube-api-access-hk4dl\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:26.154825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.154612    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33b4f43c-0281-4152-9082-6460ccbfe0fa-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:26.154825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.154625    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b4f43c-0281-4152-9082-6460ccbfe0fa-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:03:26.316194 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.316110    2560 generic.go:356] "Generic (PLEG): container finished" podID="33b4f43c-0281-4152-9082-6460ccbfe0fa" containerID="d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd" exitCode=0
Sep 16 18:03:26.316194 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.316153    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"
Sep 16 18:03:26.316194 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.316167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6" event={"ID":"33b4f43c-0281-4152-9082-6460ccbfe0fa","Type":"ContainerDied","Data":"d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd"}
Sep 16 18:03:26.316442 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.316207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6" event={"ID":"33b4f43c-0281-4152-9082-6460ccbfe0fa","Type":"ContainerDied","Data":"a7e1bf2c45260cd5731da00db11b71897777190160ec58f8cb139920f1761e49"}
Sep 16 18:03:26.316442 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.316237    2560 scope.go:117] "RemoveContainer" containerID="d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd"
Sep 16 18:03:26.324855 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.324838    2560 scope.go:117] "RemoveContainer" containerID="d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd"
Sep 16 18:03:26.325141 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:03:26.325121    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd\": container with ID starting with d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd not found: ID does not exist" containerID="d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd"
Sep 16 18:03:26.325185 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.325150    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd"} err="failed to get container status \"d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd\": rpc error: code = NotFound desc = could not find container \"d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd\": container with ID starting with d68fbd485f04a3e505e55d99101e3b49605ad8f8bda96d8cfe0ed940cce0e8fd not found: ID does not exist"
Sep 16 18:03:26.359026 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.358998    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"]
Sep 16 18:03:26.372294 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:26.372271    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-wn6x6"]
Sep 16 18:03:27.118274 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:27.118237    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b4f43c-0281-4152-9082-6460ccbfe0fa" path="/var/lib/kubelet/pods/33b4f43c-0281-4152-9082-6460ccbfe0fa/volumes"
Sep 16 18:03:28.111361 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111321    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"]
Sep 16 18:03:28.111734 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111714    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b4f43c-0281-4152-9082-6460ccbfe0fa" containerName="main"
Sep 16 18:03:28.111734 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111735    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b4f43c-0281-4152-9082-6460ccbfe0fa" containerName="main"
Sep 16 18:03:28.111902 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111760    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e663e6df-0d61-4412-a827-c8b9e883e20c" containerName="main"
Sep 16 18:03:28.111902 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111769    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e663e6df-0d61-4412-a827-c8b9e883e20c" containerName="main"
Sep 16 18:03:28.111902 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111849    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b4f43c-0281-4152-9082-6460ccbfe0fa" containerName="main"
Sep 16 18:03:28.111902 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.111866    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e663e6df-0d61-4412-a827-c8b9e883e20c" containerName="main"
Sep 16 18:03:28.114713 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.114692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.124298 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.124279    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Sep 16 18:03:28.156734 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.156704    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"]
Sep 16 18:03:28.172266 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.172240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-model-cache\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.172372 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.172279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.172372 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.172333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-dshm\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.172372 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.172350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jhq\" (UniqueName: \"kubernetes.io/projected/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-kube-api-access-79jhq\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.172524 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.172435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-home\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.172524 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.172465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273430 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273397    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-home\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273430 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273430    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273679 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-model-cache\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273679 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273679 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-dshm\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273679 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jhq\" (UniqueName: \"kubernetes.io/projected/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-kube-api-access-79jhq\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273897 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-home\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.273953 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-model-cache\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.274006 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.273986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.275770 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.275752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-dshm\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.276170 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.276151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.287492 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.287468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jhq\" (UniqueName: \"kubernetes.io/projected/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-kube-api-access-79jhq\") pod \"tri-v2-kserve-dcb554bf5-d9vwg\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.424235 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.424130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:28.706970 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:28.706939    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"]
Sep 16 18:03:28.710147 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:03:28.710122    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29c1f11_4c50_4f67_8c0b_addbb733a3a7.slice/crio-54bea4f2492ae67dd52d9e0a0f7c395272e45303ca53cbe1f59cac1d2fbedd90 WatchSource:0}: Error finding container 54bea4f2492ae67dd52d9e0a0f7c395272e45303ca53cbe1f59cac1d2fbedd90: Status 404 returned error can't find the container with id 54bea4f2492ae67dd52d9e0a0f7c395272e45303ca53cbe1f59cac1d2fbedd90
Sep 16 18:03:29.331264 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:29.331222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg" event={"ID":"b29c1f11-4c50-4f67-8c0b-addbb733a3a7","Type":"ContainerStarted","Data":"3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7"}
Sep 16 18:03:29.331668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:29.331274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg" event={"ID":"b29c1f11-4c50-4f67-8c0b-addbb733a3a7","Type":"ContainerStarted","Data":"54bea4f2492ae67dd52d9e0a0f7c395272e45303ca53cbe1f59cac1d2fbedd90"}
Sep 16 18:03:29.355270 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:29.355220    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg" podStartSLOduration=1.35520565 podStartE2EDuration="1.35520565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:03:29.35381018 +0000 UTC m=+2324.762927428" watchObservedRunningTime="2026-09-16 18:03:29.35520565 +0000 UTC m=+2324.764322898"
Sep 16 18:03:29.424686 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:29.424658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:38.424424 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:38.424340    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:38.436566 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:38.436541    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:03:39.387793 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:03:39.387761    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:04:11.588399 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.588357    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"]
Sep 16 18:04:11.590890 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.590871    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.594001 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.593975    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Sep 16 18:04:11.606984 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.606963    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"]
Sep 16 18:04:11.611914 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.611880    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-dshm\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.612042 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.611920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3808c8-8a82-4c51-8188-7365b416008b-tls-certs\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.612042 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.611962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-model-cache\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.612042 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.612002    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.612222 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.612093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-home\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.612222 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.612140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdjg\" (UniqueName: \"kubernetes.io/projected/1e3808c8-8a82-4c51-8188-7365b416008b-kube-api-access-dfdjg\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713237 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-dshm\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713237 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3808c8-8a82-4c51-8188-7365b416008b-tls-certs\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713467 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-model-cache\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713467 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713467 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-home\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713739 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdjg\" (UniqueName: \"kubernetes.io/projected/1e3808c8-8a82-4c51-8188-7365b416008b-kube-api-access-dfdjg\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713849 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713825    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.713912 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.713891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-model-cache\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.714186 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.714161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-home\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.715880 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.715859    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-dshm\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.716016 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.715994    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3808c8-8a82-4c51-8188-7365b416008b-tls-certs\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.723739 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.723719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdjg\" (UniqueName: \"kubernetes.io/projected/1e3808c8-8a82-4c51-8188-7365b416008b-kube-api-access-dfdjg\") pod \"tri-v3-kserve-798c88c66c-mdxbp\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:11.901472 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:11.901392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:12.178772 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:12.178692    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"]
Sep 16 18:04:12.182204 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:04:12.182176    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3808c8_8a82_4c51_8188_7365b416008b.slice/crio-c015120db2588baf6d612bed375a0c9af76af72db16540cee7989c6792f49f07 WatchSource:0}: Error finding container c015120db2588baf6d612bed375a0c9af76af72db16540cee7989c6792f49f07: Status 404 returned error can't find the container with id c015120db2588baf6d612bed375a0c9af76af72db16540cee7989c6792f49f07
Sep 16 18:04:12.512907 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:12.512863    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp" event={"ID":"1e3808c8-8a82-4c51-8188-7365b416008b","Type":"ContainerStarted","Data":"c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd"}
Sep 16 18:04:12.513117 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:12.512935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp" event={"ID":"1e3808c8-8a82-4c51-8188-7365b416008b","Type":"ContainerStarted","Data":"c015120db2588baf6d612bed375a0c9af76af72db16540cee7989c6792f49f07"}
Sep 16 18:04:12.540236 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:12.540155    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp" podStartSLOduration=1.540132484 podStartE2EDuration="1.540132484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:12.537294717 +0000 UTC m=+2367.946411966" watchObservedRunningTime="2026-09-16 18:04:12.540132484 +0000 UTC m=+2367.949249733"
Sep 16 18:04:12.901940 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:12.901855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:21.902265 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:21.902226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:21.914204 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:21.914182    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:22.565365 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:22.565337    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:04:54.724378 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.724343    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"]
Sep 16 18:04:54.726782 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.726760    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.733851 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.733829    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 18:04:54.733952 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.733865    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 18:04:54.734815 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.734795    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:04:54.734890 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.734805    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-rhvtt\""
Sep 16 18:04:54.734950 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.734806    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 18:04:54.735897 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.735883    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 16 18:04:54.745042 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.745019    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"]
Sep 16 18:04:54.867678 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-dshm\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-home\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867960 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfk9\" (UniqueName: \"kubernetes.io/projected/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kube-api-access-9qfk9\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867960 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867871    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867960 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e481c2c-bc41-404a-bebf-ab2a7536d54a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.867960 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.867929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.968811 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e481c2c-bc41-404a-bebf-ab2a7536d54a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.968811 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-dshm\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-home\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.968988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfk9\" (UniqueName: \"kubernetes.io/projected/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kube-api-access-9qfk9\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.969013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969352 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.969328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-home\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969417 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.969390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969473 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.969438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-model-cache\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969570 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.969549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e481c2c-bc41-404a-bebf-ab2a7536d54a-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.969639 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.969613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.971147 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.971121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-dshm\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.971450 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.971434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:54.982479 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:54.982423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfk9\" (UniqueName: \"kubernetes.io/projected/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kube-api-access-9qfk9\") pod \"tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:55.036975 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:55.036949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:04:55.333385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:55.333304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"]
Sep 16 18:04:55.337694 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:04:55.337659    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e481c2c_bc41_404a_bebf_ab2a7536d54a.slice/crio-07457a3303017ae49fad874b017a0bb0f47d7e4a8f4af89dd967e3dfe58706ec WatchSource:0}: Error finding container 07457a3303017ae49fad874b017a0bb0f47d7e4a8f4af89dd967e3dfe58706ec: Status 404 returned error can't find the container with id 07457a3303017ae49fad874b017a0bb0f47d7e4a8f4af89dd967e3dfe58706ec
Sep 16 18:04:55.693199 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:55.693101    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" event={"ID":"4e481c2c-bc41-404a-bebf-ab2a7536d54a","Type":"ContainerStarted","Data":"ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d"}
Sep 16 18:04:55.693199 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:55.693165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" event={"ID":"4e481c2c-bc41-404a-bebf-ab2a7536d54a","Type":"ContainerStarted","Data":"07457a3303017ae49fad874b017a0bb0f47d7e4a8f4af89dd967e3dfe58706ec"}
Sep 16 18:04:57.703620 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:57.703572    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerID="ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d" exitCode=0
Sep 16 18:04:57.703929 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:57.703649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" event={"ID":"4e481c2c-bc41-404a-bebf-ab2a7536d54a","Type":"ContainerDied","Data":"ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d"}
Sep 16 18:04:58.710184 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:58.710149    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" event={"ID":"4e481c2c-bc41-404a-bebf-ab2a7536d54a","Type":"ContainerStarted","Data":"f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"}
Sep 16 18:04:58.793173 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:58.793117    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" podStartSLOduration=2.5472726 podStartE2EDuration="4.79310025s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.24582765s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:04:58.791668931 +0000 UTC m=+2414.200786179" watchObservedRunningTime="2026-09-16 18:04:58.79310025 +0000 UTC m=+2414.202217500"
Sep 16 18:04:59.037291 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:04:59.037212    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:05:05.037835 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:05.037791    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:05:05.050651 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:05.050623    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:05:05.751429 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:05.751396    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:05:39.523924 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.523890    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"]
Sep 16 18:05:39.532928 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:05:39.532889    2560 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:05:39.532928 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:05:39.532928 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:05:39.532928 ip-10-0-128-156 kubenswrapper[2560]:  > containerID="f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262" cmd=["/bin/sleep","15"]
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.532944    2560 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" message="" err=<
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]:  >
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:05:39.532958    2560 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="main" containerID="cri-o://f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"
Sep 16 18:05:39.533259 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.533007    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="main" containerID="cri-o://f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262" gracePeriod=60
Sep 16 18:05:39.740861 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.740826    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Sep 16 18:05:39.873990 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.873962    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:05:39.886439 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.886415    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerID="f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262" exitCode=0
Sep 16 18:05:39.886544 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.886449    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"
Sep 16 18:05:39.886544 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.886460    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" event={"ID":"4e481c2c-bc41-404a-bebf-ab2a7536d54a","Type":"ContainerDied","Data":"f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"}
Sep 16 18:05:39.886544 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.886500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8" event={"ID":"4e481c2c-bc41-404a-bebf-ab2a7536d54a","Type":"ContainerDied","Data":"07457a3303017ae49fad874b017a0bb0f47d7e4a8f4af89dd967e3dfe58706ec"}
Sep 16 18:05:39.886544 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.886520    2560 scope.go:117] "RemoveContainer" containerID="f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"
Sep 16 18:05:39.894531 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.894514    2560 scope.go:117] "RemoveContainer" containerID="ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d"
Sep 16 18:05:39.901676 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.901578    2560 scope.go:117] "RemoveContainer" containerID="f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"
Sep 16 18:05:39.901928 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:05:39.901911    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262\": container with ID starting with f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262 not found: ID does not exist" containerID="f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"
Sep 16 18:05:39.901974 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.901933    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262"} err="failed to get container status \"f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262\": rpc error: code = NotFound desc = could not find container \"f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262\": container with ID starting with f2e66a02497e529a8ac6f3b48537a02b9da5ef674f43d340cf5811f9cbf0c262 not found: ID does not exist"
Sep 16 18:05:39.901974 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.901948    2560 scope.go:117] "RemoveContainer" containerID="ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d"
Sep 16 18:05:39.902207 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:05:39.902190    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d\": container with ID starting with ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d not found: ID does not exist" containerID="ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d"
Sep 16 18:05:39.902260 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.902210    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d"} err="failed to get container status \"ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d\": rpc error: code = NotFound desc = could not find container \"ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d\": container with ID starting with ec37e4bb376372fcf136b6e4f18ff995247a19bd850219638f4ee91be4c1662d not found: ID does not exist"
Sep 16 18:05:39.961367 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e481c2c-bc41-404a-bebf-ab2a7536d54a-cabundle-cert\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961374    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qfk9\" (UniqueName: \"kubernetes.io/projected/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kube-api-access-9qfk9\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961414    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tls-certs\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961436    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-dshm\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961465    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-model-cache\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kserve-provision-location\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961534    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tmp-dir\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961916 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961605    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-home\") pod \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\" (UID: \"4e481c2c-bc41-404a-bebf-ab2a7536d54a\") "
Sep 16 18:05:39.961916 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961834    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.961916 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.961866    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e481c2c-bc41-404a-bebf-ab2a7536d54a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:05:39.962078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.962018    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-home" (OuterVolumeSpecName: "home") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.962078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.962053    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.963825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.963798    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:05:39.963933 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.963844    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kube-api-access-9qfk9" (OuterVolumeSpecName: "kube-api-access-9qfk9") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "kube-api-access-9qfk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:05:39.963933 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.963915    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-dshm" (OuterVolumeSpecName: "dshm") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:39.982285 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:39.982251    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e481c2c-bc41-404a-bebf-ab2a7536d54a" (UID: "4e481c2c-bc41-404a-bebf-ab2a7536d54a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063073    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063100    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063111    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063120    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063129    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063139    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e481c2c-bc41-404a-bebf-ab2a7536d54a-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063147    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e481c2c-bc41-404a-bebf-ab2a7536d54a-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.063154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.063155    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfk9\" (UniqueName: \"kubernetes.io/projected/4e481c2c-bc41-404a-bebf-ab2a7536d54a-kube-api-access-9qfk9\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:05:40.212394 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.212362    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"]
Sep 16 18:05:40.218922 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:40.218896    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6c5fc8fb79-pgwm8"]
Sep 16 18:05:41.118431 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:05:41.118389    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" path="/var/lib/kubelet/pods/4e481c2c-bc41-404a-bebf-ab2a7536d54a/volumes"
Sep 16 18:06:03.746992 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:03.746956    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"]
Sep 16 18:06:04.914609 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:04.914368    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"]
Sep 16 18:06:05.647499 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:05.647468    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"]
Sep 16 18:06:18.765686 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:18.765647    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp" podUID="1e3808c8-8a82-4c51-8188-7365b416008b" containerName="main" containerID="cri-o://c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd" gracePeriod=45
Sep 16 18:06:19.005689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.005662    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:06:19.047250 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.047160    2560 generic.go:356] "Generic (PLEG): container finished" podID="1e3808c8-8a82-4c51-8188-7365b416008b" containerID="c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd" exitCode=0
Sep 16 18:06:19.047250 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.047209    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"
Sep 16 18:06:19.047250 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.047237    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp" event={"ID":"1e3808c8-8a82-4c51-8188-7365b416008b","Type":"ContainerDied","Data":"c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd"}
Sep 16 18:06:19.047512 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.047290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp" event={"ID":"1e3808c8-8a82-4c51-8188-7365b416008b","Type":"ContainerDied","Data":"c015120db2588baf6d612bed375a0c9af76af72db16540cee7989c6792f49f07"}
Sep 16 18:06:19.047512 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.047315    2560 scope.go:117] "RemoveContainer" containerID="c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd"
Sep 16 18:06:19.055453 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.055434    2560 scope.go:117] "RemoveContainer" containerID="c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd"
Sep 16 18:06:19.055739 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:06:19.055713    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd\": container with ID starting with c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd not found: ID does not exist" containerID="c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd"
Sep 16 18:06:19.055818 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.055749    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd"} err="failed to get container status \"c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd\": rpc error: code = NotFound desc = could not find container \"c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd\": container with ID starting with c69af1c61c4641e197d64aed70aa68edc7544b02f92bd206ee8ee62e01a1ddfd not found: ID does not exist"
Sep 16 18:06:19.089088 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089058    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-model-cache\") pod \"1e3808c8-8a82-4c51-8188-7365b416008b\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") "
Sep 16 18:06:19.089088 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089091    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3808c8-8a82-4c51-8188-7365b416008b-tls-certs\") pod \"1e3808c8-8a82-4c51-8188-7365b416008b\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") "
Sep 16 18:06:19.089239 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-dshm\") pod \"1e3808c8-8a82-4c51-8188-7365b416008b\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") "
Sep 16 18:06:19.089239 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089155    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-home\") pod \"1e3808c8-8a82-4c51-8188-7365b416008b\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") "
Sep 16 18:06:19.089239 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfdjg\" (UniqueName: \"kubernetes.io/projected/1e3808c8-8a82-4c51-8188-7365b416008b-kube-api-access-dfdjg\") pod \"1e3808c8-8a82-4c51-8188-7365b416008b\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") "
Sep 16 18:06:19.089239 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-tmp-dir\") pod \"1e3808c8-8a82-4c51-8188-7365b416008b\" (UID: \"1e3808c8-8a82-4c51-8188-7365b416008b\") "
Sep 16 18:06:19.089425 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089367    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e3808c8-8a82-4c51-8188-7365b416008b" (UID: "1e3808c8-8a82-4c51-8188-7365b416008b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:19.089532 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-home" (OuterVolumeSpecName: "home") pod "1e3808c8-8a82-4c51-8188-7365b416008b" (UID: "1e3808c8-8a82-4c51-8188-7365b416008b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:19.089532 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089508    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:19.089678 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.089541    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e3808c8-8a82-4c51-8188-7365b416008b" (UID: "1e3808c8-8a82-4c51-8188-7365b416008b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:19.091318 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.091289    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e3808c8-8a82-4c51-8188-7365b416008b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e3808c8-8a82-4c51-8188-7365b416008b" (UID: "1e3808c8-8a82-4c51-8188-7365b416008b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:06:19.091416 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.091347    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-dshm" (OuterVolumeSpecName: "dshm") pod "1e3808c8-8a82-4c51-8188-7365b416008b" (UID: "1e3808c8-8a82-4c51-8188-7365b416008b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:19.091416 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.091402    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e3808c8-8a82-4c51-8188-7365b416008b-kube-api-access-dfdjg" (OuterVolumeSpecName: "kube-api-access-dfdjg") pod "1e3808c8-8a82-4c51-8188-7365b416008b" (UID: "1e3808c8-8a82-4c51-8188-7365b416008b"). InnerVolumeSpecName "kube-api-access-dfdjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:06:19.190052 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.190020    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e3808c8-8a82-4c51-8188-7365b416008b-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:19.190052 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.190048    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:19.190230 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.190061    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:19.190230 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.190073    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdjg\" (UniqueName: \"kubernetes.io/projected/1e3808c8-8a82-4c51-8188-7365b416008b-kube-api-access-dfdjg\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:19.190230 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.190086    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e3808c8-8a82-4c51-8188-7365b416008b-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:19.369789 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.369714    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"]
Sep 16 18:06:19.378385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.378358    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-mdxbp"]
Sep 16 18:06:19.934991 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:19.934933    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p" podUID="c46d4737-9c08-423f-88b6-d5ee457bba4b" containerName="main" containerID="cri-o://bca67ecc018a77694955789869df925c76bd8a5ff6b9553d7e4102360d7de09a" gracePeriod=45
Sep 16 18:06:20.053366 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.053339    2560 generic.go:356] "Generic (PLEG): container finished" podID="c46d4737-9c08-423f-88b6-d5ee457bba4b" containerID="bca67ecc018a77694955789869df925c76bd8a5ff6b9553d7e4102360d7de09a" exitCode=0
Sep 16 18:06:20.053495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.053374    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p" event={"ID":"c46d4737-9c08-423f-88b6-d5ee457bba4b","Type":"ContainerDied","Data":"bca67ecc018a77694955789869df925c76bd8a5ff6b9553d7e4102360d7de09a"}
Sep 16 18:06:20.180050 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.180027    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:06:20.298496 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298467    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46d4737-9c08-423f-88b6-d5ee457bba4b-tls-certs\") pod \"c46d4737-9c08-423f-88b6-d5ee457bba4b\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") "
Sep 16 18:06:20.298661 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298505    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htwz7\" (UniqueName: \"kubernetes.io/projected/c46d4737-9c08-423f-88b6-d5ee457bba4b-kube-api-access-htwz7\") pod \"c46d4737-9c08-423f-88b6-d5ee457bba4b\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") "
Sep 16 18:06:20.298661 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298536    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-tmp-dir\") pod \"c46d4737-9c08-423f-88b6-d5ee457bba4b\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") "
Sep 16 18:06:20.298661 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298633    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-model-cache\") pod \"c46d4737-9c08-423f-88b6-d5ee457bba4b\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") "
Sep 16 18:06:20.298661 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298657    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-home\") pod \"c46d4737-9c08-423f-88b6-d5ee457bba4b\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") "
Sep 16 18:06:20.298875 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298679    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-dshm\") pod \"c46d4737-9c08-423f-88b6-d5ee457bba4b\" (UID: \"c46d4737-9c08-423f-88b6-d5ee457bba4b\") "
Sep 16 18:06:20.298875 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298808    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c46d4737-9c08-423f-88b6-d5ee457bba4b" (UID: "c46d4737-9c08-423f-88b6-d5ee457bba4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:20.298986 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298940    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:20.298986 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298950    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c46d4737-9c08-423f-88b6-d5ee457bba4b" (UID: "c46d4737-9c08-423f-88b6-d5ee457bba4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:20.298986 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.298968    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-home" (OuterVolumeSpecName: "home") pod "c46d4737-9c08-423f-88b6-d5ee457bba4b" (UID: "c46d4737-9c08-423f-88b6-d5ee457bba4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:20.300724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.300701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46d4737-9c08-423f-88b6-d5ee457bba4b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c46d4737-9c08-423f-88b6-d5ee457bba4b" (UID: "c46d4737-9c08-423f-88b6-d5ee457bba4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:06:20.300792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.300732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-dshm" (OuterVolumeSpecName: "dshm") pod "c46d4737-9c08-423f-88b6-d5ee457bba4b" (UID: "c46d4737-9c08-423f-88b6-d5ee457bba4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:20.300842 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.300784    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46d4737-9c08-423f-88b6-d5ee457bba4b-kube-api-access-htwz7" (OuterVolumeSpecName: "kube-api-access-htwz7") pod "c46d4737-9c08-423f-88b6-d5ee457bba4b" (UID: "c46d4737-9c08-423f-88b6-d5ee457bba4b"). InnerVolumeSpecName "kube-api-access-htwz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:06:20.399527 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.399490    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:20.399527 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.399521    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:20.399527 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.399529    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46d4737-9c08-423f-88b6-d5ee457bba4b-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:20.399751 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.399537    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46d4737-9c08-423f-88b6-d5ee457bba4b-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:20.399751 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:20.399546    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwz7\" (UniqueName: \"kubernetes.io/projected/c46d4737-9c08-423f-88b6-d5ee457bba4b-kube-api-access-htwz7\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:21.008723 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.008678    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg" podUID="b29c1f11-4c50-4f67-8c0b-addbb733a3a7" containerName="main" containerID="cri-o://3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7" gracePeriod=45
Sep 16 18:06:21.059056 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.059032    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"
Sep 16 18:06:21.059160 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.059050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p" event={"ID":"c46d4737-9c08-423f-88b6-d5ee457bba4b","Type":"ContainerDied","Data":"7ca65cde88079a79475dc6a58b28b7267216e0ee379db9ce0221e41b258c5334"}
Sep 16 18:06:21.059160 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.059105    2560 scope.go:117] "RemoveContainer" containerID="bca67ecc018a77694955789869df925c76bd8a5ff6b9553d7e4102360d7de09a"
Sep 16 18:06:21.090176 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.090153    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"]
Sep 16 18:06:21.094890 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.094870    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v1-kserve-79f8f98bc5-w9k4p"]
Sep 16 18:06:21.117262 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.117237    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3808c8-8a82-4c51-8188-7365b416008b" path="/var/lib/kubelet/pods/1e3808c8-8a82-4c51-8188-7365b416008b/volumes"
Sep 16 18:06:21.117733 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.117713    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c46d4737-9c08-423f-88b6-d5ee457bba4b" path="/var/lib/kubelet/pods/c46d4737-9c08-423f-88b6-d5ee457bba4b/volumes"
Sep 16 18:06:21.259548 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.259484    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:06:21.407654 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.407622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-dshm\") pod \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") "
Sep 16 18:06:21.407828 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.407661    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79jhq\" (UniqueName: \"kubernetes.io/projected/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-kube-api-access-79jhq\") pod \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") "
Sep 16 18:06:21.407828 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.407686    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tmp-dir\") pod \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") "
Sep 16 18:06:21.407828 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.407729    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-home\") pod \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") "
Sep 16 18:06:21.407828 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.407763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tls-certs\") pod \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") "
Sep 16 18:06:21.407828 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.407794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-model-cache\") pod \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\" (UID: \"b29c1f11-4c50-4f67-8c0b-addbb733a3a7\") "
Sep 16 18:06:21.408090 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.408013    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b29c1f11-4c50-4f67-8c0b-addbb733a3a7" (UID: "b29c1f11-4c50-4f67-8c0b-addbb733a3a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:21.408090 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.408028    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-home" (OuterVolumeSpecName: "home") pod "b29c1f11-4c50-4f67-8c0b-addbb733a3a7" (UID: "b29c1f11-4c50-4f67-8c0b-addbb733a3a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:21.408170 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.408099    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b29c1f11-4c50-4f67-8c0b-addbb733a3a7" (UID: "b29c1f11-4c50-4f67-8c0b-addbb733a3a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:21.409872 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.409851    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-kube-api-access-79jhq" (OuterVolumeSpecName: "kube-api-access-79jhq") pod "b29c1f11-4c50-4f67-8c0b-addbb733a3a7" (UID: "b29c1f11-4c50-4f67-8c0b-addbb733a3a7"). InnerVolumeSpecName "kube-api-access-79jhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:06:21.409965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.409906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b29c1f11-4c50-4f67-8c0b-addbb733a3a7" (UID: "b29c1f11-4c50-4f67-8c0b-addbb733a3a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:06:21.409965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.409947    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-dshm" (OuterVolumeSpecName: "dshm") pod "b29c1f11-4c50-4f67-8c0b-addbb733a3a7" (UID: "b29c1f11-4c50-4f67-8c0b-addbb733a3a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:06:21.508911 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.508886    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:21.508911 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.508908    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:21.509047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.508919    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:21.509047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.508928    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:21.509047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.508937    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jhq\" (UniqueName: \"kubernetes.io/projected/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-kube-api-access-79jhq\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:21.509047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:21.508945    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29c1f11-4c50-4f67-8c0b-addbb733a3a7-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:06:22.064331 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.064296    2560 generic.go:356] "Generic (PLEG): container finished" podID="b29c1f11-4c50-4f67-8c0b-addbb733a3a7" containerID="3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7" exitCode=0
Sep 16 18:06:22.064794 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.064358    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"
Sep 16 18:06:22.064794 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.064370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg" event={"ID":"b29c1f11-4c50-4f67-8c0b-addbb733a3a7","Type":"ContainerDied","Data":"3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7"}
Sep 16 18:06:22.064794 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.064453    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg" event={"ID":"b29c1f11-4c50-4f67-8c0b-addbb733a3a7","Type":"ContainerDied","Data":"54bea4f2492ae67dd52d9e0a0f7c395272e45303ca53cbe1f59cac1d2fbedd90"}
Sep 16 18:06:22.064794 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.064488    2560 scope.go:117] "RemoveContainer" containerID="3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7"
Sep 16 18:06:22.073351 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.073333    2560 scope.go:117] "RemoveContainer" containerID="3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7"
Sep 16 18:06:22.073612 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:06:22.073577    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7\": container with ID starting with 3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7 not found: ID does not exist" containerID="3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7"
Sep 16 18:06:22.073668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.073619    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7"} err="failed to get container status \"3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7\": rpc error: code = NotFound desc = could not find container \"3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7\": container with ID starting with 3cffff31721dd6a2f138495d52b0e4bcd7ee69ea35fdbed27fe781cc9c1a48f7 not found: ID does not exist"
Sep 16 18:06:22.085986 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.085965    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"]
Sep 16 18:06:22.090230 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:22.090210    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-d9vwg"]
Sep 16 18:06:23.116913 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:23.116881    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b29c1f11-4c50-4f67-8c0b-addbb733a3a7" path="/var/lib/kubelet/pods/b29c1f11-4c50-4f67-8c0b-addbb733a3a7/volumes"
Sep 16 18:06:26.169105 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169074    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"]
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169362    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="storage-initializer"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169374    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="storage-initializer"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169392    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46d4737-9c08-423f-88b6-d5ee457bba4b" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169397    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46d4737-9c08-423f-88b6-d5ee457bba4b" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169405    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3808c8-8a82-4c51-8188-7365b416008b" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169410    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3808c8-8a82-4c51-8188-7365b416008b" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169419    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29c1f11-4c50-4f67-8c0b-addbb733a3a7" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169424    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29c1f11-4c50-4f67-8c0b-addbb733a3a7" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169432    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="main"
Sep 16 18:06:26.169468 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169437    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="main"
Sep 16 18:06:26.169835 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169482    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46d4737-9c08-423f-88b6-d5ee457bba4b" containerName="main"
Sep 16 18:06:26.169835 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169490    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b29c1f11-4c50-4f67-8c0b-addbb733a3a7" containerName="main"
Sep 16 18:06:26.169835 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169498    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e481c2c-bc41-404a-bebf-ab2a7536d54a" containerName="main"
Sep 16 18:06:26.169835 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.169504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3808c8-8a82-4c51-8188-7365b416008b" containerName="main"
Sep 16 18:06:26.171572 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.171554    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.174729 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.174705    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 18:06:26.175262 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.175239    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 18:06:26.175385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.175261    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 18:06:26.175385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.175299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-l4pm2\""
Sep 16 18:06:26.186516 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.186493    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"]
Sep 16 18:06:26.246453 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.246421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tls-certs\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.246654 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.246460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2gb\" (UniqueName: \"kubernetes.io/projected/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-kube-api-access-wm2gb\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.246654 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.246486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-home\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.246654 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.246527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-dshm\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.246654 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.246607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.246654 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.246652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-model-cache\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347395 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tls-certs\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347395 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2gb\" (UniqueName: \"kubernetes.io/projected/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-kube-api-access-wm2gb\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347676 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-home\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347676 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-dshm\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347676 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347863 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-model-cache\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347922 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347895    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-home\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.347973 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.347949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.348062 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.348042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-model-cache\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.349553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.349534    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-dshm\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.349929 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.349913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tls-certs\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.357460 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.357441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2gb\" (UniqueName: \"kubernetes.io/projected/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-kube-api-access-wm2gb\") pod \"late-v1-kserve-56cbc8456c-rqh48\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.481269 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.481230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:26.790675 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:26.790550    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"]
Sep 16 18:06:26.794351 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:06:26.794324    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b23e51_e7cc_4028_baf6_9f36c71d2ebe.slice/crio-7a1f4254e1073419d2e3e5b0a9cf477286e56422b5b9f900138f66ca59a1e90f WatchSource:0}: Error finding container 7a1f4254e1073419d2e3e5b0a9cf477286e56422b5b9f900138f66ca59a1e90f: Status 404 returned error can't find the container with id 7a1f4254e1073419d2e3e5b0a9cf477286e56422b5b9f900138f66ca59a1e90f
Sep 16 18:06:27.092754 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:27.092663    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48" event={"ID":"53b23e51-e7cc-4028-baf6-9f36c71d2ebe","Type":"ContainerStarted","Data":"42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf"}
Sep 16 18:06:27.092908 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:27.092758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48" event={"ID":"53b23e51-e7cc-4028-baf6-9f36c71d2ebe","Type":"ContainerStarted","Data":"7a1f4254e1073419d2e3e5b0a9cf477286e56422b5b9f900138f66ca59a1e90f"}
Sep 16 18:06:27.120255 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:27.120191    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48" podStartSLOduration=1.120176294 podStartE2EDuration="1.120176294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:06:27.117976724 +0000 UTC m=+2502.527093973" watchObservedRunningTime="2026-09-16 18:06:27.120176294 +0000 UTC m=+2502.529293543"
Sep 16 18:06:27.481343 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:27.481311    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:36.482103 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:36.482065    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:36.494249 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:36.494223    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:37.143406 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:37.143376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:06:49.395045 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.395009    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"]
Sep 16 18:06:49.398421 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.398403    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.401379 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.401359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 18:06:49.401754 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.401728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 18:06:49.401754 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.401745    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-q6qv9\""
Sep 16 18:06:49.401948 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.401749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 16 18:06:49.401948 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.401750    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:06:49.401948 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.401770    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 18:06:49.424240 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.424208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"]
Sep 16 18:06:49.433931 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.433902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434038 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.433979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-dshm\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434038 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.434006    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20343c9a-c0a4-4800-99a6-7d3d97e65820-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434128 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.434035    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48k9m\" (UniqueName: \"kubernetes.io/projected/20343c9a-c0a4-4800-99a6-7d3d97e65820-kube-api-access-48k9m\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434128 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.434093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-model-cache\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434128 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.434117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434250 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.434147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-home\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.434250 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.434184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20343c9a-c0a4-4800-99a6-7d3d97e65820-tls-certs\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535197 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-home\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20343c9a-c0a4-4800-99a6-7d3d97e65820-tls-certs\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-dshm\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20343c9a-c0a4-4800-99a6-7d3d97e65820-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48k9m\" (UniqueName: \"kubernetes.io/projected/20343c9a-c0a4-4800-99a6-7d3d97e65820-kube-api-access-48k9m\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535349    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-model-cache\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535726 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535783 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535728    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-home\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535890 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-model-cache\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.535966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.535935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.536201 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.536180    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20343c9a-c0a4-4800-99a6-7d3d97e65820-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.537659 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.537637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-dshm\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.537923 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.537908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20343c9a-c0a4-4800-99a6-7d3d97e65820-tls-certs\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.549369 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.549347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48k9m\" (UniqueName: \"kubernetes.io/projected/20343c9a-c0a4-4800-99a6-7d3d97e65820-kube-api-access-48k9m\") pod \"tokenizer-template-args-test-kserve-5968d8df69-cs9wq\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.708167 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.708141    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:49.858062 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.858028    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"]
Sep 16 18:06:49.860689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.860671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:49.863282 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.863255    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-98vxl\""
Sep 16 18:06:49.873636 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.873614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"]
Sep 16 18:06:49.939212 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.939180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rv4\" (UniqueName: \"kubernetes.io/projected/d26334f7-7631-47ce-8a86-abd30132fd62-kube-api-access-k4rv4\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:49.939345 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:49.939223    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26334f7-7631-47ce-8a86-abd30132fd62-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:50.005889 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.005821    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"]
Sep 16 18:06:50.008915 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:06:50.008889    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20343c9a_c0a4_4800_99a6_7d3d97e65820.slice/crio-1583ac76e83c896bc219a88a6c63b88975d46f496930660f19575a953dcf87a4 WatchSource:0}: Error finding container 1583ac76e83c896bc219a88a6c63b88975d46f496930660f19575a953dcf87a4: Status 404 returned error can't find the container with id 1583ac76e83c896bc219a88a6c63b88975d46f496930660f19575a953dcf87a4
Sep 16 18:06:50.040015 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.039992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rv4\" (UniqueName: \"kubernetes.io/projected/d26334f7-7631-47ce-8a86-abd30132fd62-kube-api-access-k4rv4\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:50.040138 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.040028    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26334f7-7631-47ce-8a86-abd30132fd62-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:50.042382 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.042362    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26334f7-7631-47ce-8a86-abd30132fd62-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:50.048440 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.048422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rv4\" (UniqueName: \"kubernetes.io/projected/d26334f7-7631-47ce-8a86-abd30132fd62-kube-api-access-k4rv4\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:50.170612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.170559    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:50.182000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.181972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" event={"ID":"20343c9a-c0a4-4800-99a6-7d3d97e65820","Type":"ContainerStarted","Data":"f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5"}
Sep 16 18:06:50.182133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.182032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" event={"ID":"20343c9a-c0a4-4800-99a6-7d3d97e65820","Type":"ContainerStarted","Data":"1583ac76e83c896bc219a88a6c63b88975d46f496930660f19575a953dcf87a4"}
Sep 16 18:06:50.458421 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:50.456081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"]
Sep 16 18:06:50.463913 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:06:50.463870    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26334f7_7631_47ce_8a86_abd30132fd62.slice/crio-b4dd3a77aeb09a97d32c12d22c5b79f049a998941f63f5e2d2485e64c606dbe1 WatchSource:0}: Error finding container b4dd3a77aeb09a97d32c12d22c5b79f049a998941f63f5e2d2485e64c606dbe1: Status 404 returned error can't find the container with id b4dd3a77aeb09a97d32c12d22c5b79f049a998941f63f5e2d2485e64c606dbe1
Sep 16 18:06:51.187405 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:51.187367    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" event={"ID":"d26334f7-7631-47ce-8a86-abd30132fd62","Type":"ContainerStarted","Data":"15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1"}
Sep 16 18:06:51.187627 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:51.187430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" event={"ID":"d26334f7-7631-47ce-8a86-abd30132fd62","Type":"ContainerStarted","Data":"b4dd3a77aeb09a97d32c12d22c5b79f049a998941f63f5e2d2485e64c606dbe1"}
Sep 16 18:06:51.187627 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:51.187557    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:06:51.215662 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:51.215565    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podStartSLOduration=2.215547151 podStartE2EDuration="2.215547151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:06:51.211697752 +0000 UTC m=+2526.620815001" watchObservedRunningTime="2026-09-16 18:06:51.215547151 +0000 UTC m=+2526.624664400"
Sep 16 18:06:53.195678 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:53.195646    2560 generic.go:356] "Generic (PLEG): container finished" podID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerID="f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5" exitCode=0
Sep 16 18:06:53.196047 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:53.195698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" event={"ID":"20343c9a-c0a4-4800-99a6-7d3d97e65820","Type":"ContainerDied","Data":"f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5"}
Sep 16 18:06:54.201712 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:54.201677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" event={"ID":"20343c9a-c0a4-4800-99a6-7d3d97e65820","Type":"ContainerStarted","Data":"2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"}
Sep 16 18:06:54.241622 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:54.241554    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" podStartSLOduration=2.907199745 podStartE2EDuration="5.241536351s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.334336606s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:06:54.241091885 +0000 UTC m=+2529.650209134" watchObservedRunningTime="2026-09-16 18:06:54.241536351 +0000 UTC m=+2529.650653598"
Sep 16 18:06:54.709210 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:54.709181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:59.708504 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:59.708468    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:06:59.721046 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:06:59.721017    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:07:00.236496 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:07:00.236466    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:07:22.192420 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:07:22.192390    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:08:07.136141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.136105    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"]
Sep 16 18:08:07.140575 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.140543    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"]
Sep 16 18:08:07.149245 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:08:07.149210    2560 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:08:07.149245 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:08:07.149245 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:08:07.149245 ip-10-0-128-156 kubenswrapper[2560]:  > containerID="2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea" cmd=["/bin/sleep","15"]
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.149253    2560 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" message="" err=<
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]:  >
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:08:07.149267    2560 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="main" containerID="cri-o://2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"
Sep 16 18:08:07.149492 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.149296    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="main" containerID="cri-o://2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea" gracePeriod=60
Sep 16 18:08:07.226040 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.225954    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 16 18:08:07.493681 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.493654    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:08:07.495254 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.495231    2560 generic.go:356] "Generic (PLEG): container finished" podID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerID="2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea" exitCode=0
Sep 16 18:08:07.495362 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.495286    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" event={"ID":"20343c9a-c0a4-4800-99a6-7d3d97e65820","Type":"ContainerDied","Data":"2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"}
Sep 16 18:08:07.495362 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.495349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq" event={"ID":"20343c9a-c0a4-4800-99a6-7d3d97e65820","Type":"ContainerDied","Data":"1583ac76e83c896bc219a88a6c63b88975d46f496930660f19575a953dcf87a4"}
Sep 16 18:08:07.495450 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.495373    2560 scope.go:117] "RemoveContainer" containerID="2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"
Sep 16 18:08:07.503620 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.503582    2560 scope.go:117] "RemoveContainer" containerID="f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5"
Sep 16 18:08:07.510966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.510946    2560 scope.go:117] "RemoveContainer" containerID="2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"
Sep 16 18:08:07.511241 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:08:07.511221    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea\": container with ID starting with 2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea not found: ID does not exist" containerID="2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"
Sep 16 18:08:07.511290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.511247    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea"} err="failed to get container status \"2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea\": rpc error: code = NotFound desc = could not find container \"2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea\": container with ID starting with 2cc98775fe40e9d08b8317cff886196c180e348dbf7421c718ae809137936aea not found: ID does not exist"
Sep 16 18:08:07.511290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.511261    2560 scope.go:117] "RemoveContainer" containerID="f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5"
Sep 16 18:08:07.511551 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:08:07.511524    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5\": container with ID starting with f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5 not found: ID does not exist" containerID="f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5"
Sep 16 18:08:07.511612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.511558    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5"} err="failed to get container status \"f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5\": rpc error: code = NotFound desc = could not find container \"f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5\": container with ID starting with f6c3eb410c9762b33622fb2dfa6e8e9855b77e83ff686dc9aa346f191e010ff5 not found: ID does not exist"
Sep 16 18:08:07.576190 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576167    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-kserve-provision-location\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576298 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-dshm\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576298 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576231    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-model-cache\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576298 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-home\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576298 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576292    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-tmp-dir\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576515 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576322    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20343c9a-c0a4-4800-99a6-7d3d97e65820-cabundle-cert\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576515 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20343c9a-c0a4-4800-99a6-7d3d97e65820-tls-certs\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576515 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576384    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48k9m\" (UniqueName: \"kubernetes.io/projected/20343c9a-c0a4-4800-99a6-7d3d97e65820-kube-api-access-48k9m\") pod \"20343c9a-c0a4-4800-99a6-7d3d97e65820\" (UID: \"20343c9a-c0a4-4800-99a6-7d3d97e65820\") "
Sep 16 18:08:07.576705 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576516    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-model-cache" (OuterVolumeSpecName: "model-cache") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.576705 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576664    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.576817 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-home" (OuterVolumeSpecName: "home") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.576817 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576751    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.576817 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.576784    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20343c9a-c0a4-4800-99a6-7d3d97e65820-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:08:07.578773 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.578740    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20343c9a-c0a4-4800-99a6-7d3d97e65820-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:08:07.578881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.578810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20343c9a-c0a4-4800-99a6-7d3d97e65820-kube-api-access-48k9m" (OuterVolumeSpecName: "kube-api-access-48k9m") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "kube-api-access-48k9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:08:07.578881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.578817    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-dshm" (OuterVolumeSpecName: "dshm") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.596744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.596721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20343c9a-c0a4-4800-99a6-7d3d97e65820" (UID: "20343c9a-c0a4-4800-99a6-7d3d97e65820"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:07.677446 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677397    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.677446 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677418    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.677446 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677427    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.677446 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677435    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20343c9a-c0a4-4800-99a6-7d3d97e65820-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.677446 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677444    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20343c9a-c0a4-4800-99a6-7d3d97e65820-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.677682 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677453    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20343c9a-c0a4-4800-99a6-7d3d97e65820-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:07.677682 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:07.677461    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48k9m\" (UniqueName: \"kubernetes.io/projected/20343c9a-c0a4-4800-99a6-7d3d97e65820-kube-api-access-48k9m\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:08.500553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:08.500519    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"
Sep 16 18:08:08.540328 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:08.540297    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"]
Sep 16 18:08:08.548283 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:08.548258    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5968d8df69-cs9wq"]
Sep 16 18:08:09.117875 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:09.117841    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" path="/var/lib/kubelet/pods/20343c9a-c0a4-4800-99a6-7d3d97e65820/volumes"
Sep 16 18:08:12.192815 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:12.192771    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:08:22.156837 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:22.156798    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main" containerID="cri-o://15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1" gracePeriod=45
Sep 16 18:08:22.192725 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:22.192693    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:08:32.193224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:32.193177    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:08:32.193635 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:32.193298    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:08:34.817696 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:34.817661    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"]
Sep 16 18:08:42.192532 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:42.192488    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:08:49.836717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:49.836675    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48" podUID="53b23e51-e7cc-4028-baf6-9f36c71d2ebe" containerName="main" containerID="cri-o://42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf" gracePeriod=45
Sep 16 18:08:50.072565 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.072542    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:08:50.228247 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228219    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tls-certs\") pod \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") "
Sep 16 18:08:50.228409 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228256    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-home\") pod \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") "
Sep 16 18:08:50.228409 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228280    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-model-cache\") pod \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") "
Sep 16 18:08:50.228409 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228338    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tmp-dir\") pod \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") "
Sep 16 18:08:50.228409 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm2gb\" (UniqueName: \"kubernetes.io/projected/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-kube-api-access-wm2gb\") pod \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") "
Sep 16 18:08:50.228576 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228414    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-dshm\") pod \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\" (UID: \"53b23e51-e7cc-4028-baf6-9f36c71d2ebe\") "
Sep 16 18:08:50.228657 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228574    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-home" (OuterVolumeSpecName: "home") pod "53b23e51-e7cc-4028-baf6-9f36c71d2ebe" (UID: "53b23e51-e7cc-4028-baf6-9f36c71d2ebe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:50.228657 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228637    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53b23e51-e7cc-4028-baf6-9f36c71d2ebe" (UID: "53b23e51-e7cc-4028-baf6-9f36c71d2ebe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:50.228657 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228644    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-model-cache" (OuterVolumeSpecName: "model-cache") pod "53b23e51-e7cc-4028-baf6-9f36c71d2ebe" (UID: "53b23e51-e7cc-4028-baf6-9f36c71d2ebe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:50.228836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228721    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:50.228836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228735    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:50.228836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.228745    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:50.230525 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.230501    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-dshm" (OuterVolumeSpecName: "dshm") pod "53b23e51-e7cc-4028-baf6-9f36c71d2ebe" (UID: "53b23e51-e7cc-4028-baf6-9f36c71d2ebe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:08:50.230640 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.230620    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53b23e51-e7cc-4028-baf6-9f36c71d2ebe" (UID: "53b23e51-e7cc-4028-baf6-9f36c71d2ebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:08:50.230700 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.230681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-kube-api-access-wm2gb" (OuterVolumeSpecName: "kube-api-access-wm2gb") pod "53b23e51-e7cc-4028-baf6-9f36c71d2ebe" (UID: "53b23e51-e7cc-4028-baf6-9f36c71d2ebe"). InnerVolumeSpecName "kube-api-access-wm2gb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:08:50.330112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.330078    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:50.330112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.330103    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2gb\" (UniqueName: \"kubernetes.io/projected/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-kube-api-access-wm2gb\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:50.330112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.330113    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b23e51-e7cc-4028-baf6-9f36c71d2ebe-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:50.673624 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.673526    2560 generic.go:356] "Generic (PLEG): container finished" podID="53b23e51-e7cc-4028-baf6-9f36c71d2ebe" containerID="42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf" exitCode=0
Sep 16 18:08:50.673624 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.673571    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"
Sep 16 18:08:50.673813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.673618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48" event={"ID":"53b23e51-e7cc-4028-baf6-9f36c71d2ebe","Type":"ContainerDied","Data":"42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf"}
Sep 16 18:08:50.673813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.673675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48" event={"ID":"53b23e51-e7cc-4028-baf6-9f36c71d2ebe","Type":"ContainerDied","Data":"7a1f4254e1073419d2e3e5b0a9cf477286e56422b5b9f900138f66ca59a1e90f"}
Sep 16 18:08:50.673813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.673697    2560 scope.go:117] "RemoveContainer" containerID="42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf"
Sep 16 18:08:50.682111 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.682092    2560 scope.go:117] "RemoveContainer" containerID="42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf"
Sep 16 18:08:50.682376 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:08:50.682359    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf\": container with ID starting with 42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf not found: ID does not exist" containerID="42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf"
Sep 16 18:08:50.682423 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.682384    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf"} err="failed to get container status \"42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf\": rpc error: code = NotFound desc = could not find container \"42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf\": container with ID starting with 42ffb0bc798ea3e07b7b4999af9f809dd2997d1048295c5360534fd2cbf91cdf not found: ID does not exist"
Sep 16 18:08:50.700360 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.700332    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"]
Sep 16 18:08:50.703143 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:50.703120    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-rqh48"]
Sep 16 18:08:51.117778 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:51.117734    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b23e51-e7cc-4028-baf6-9f36c71d2ebe" path="/var/lib/kubelet/pods/53b23e51-e7cc-4028-baf6-9f36c71d2ebe/volumes"
Sep 16 18:08:52.192620 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:08:52.192579    2560 logging.go:55] [core] [Channel #425 SubChannel #426]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.51:9003", ServerName: "10.134.0.51:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.51:9003: connect: connection refused"
Sep 16 18:08:52.399354 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.399330    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:08:52.545704 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.545675    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26334f7-7631-47ce-8a86-abd30132fd62-tls-certs\") pod \"d26334f7-7631-47ce-8a86-abd30132fd62\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") "
Sep 16 18:08:52.545853 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.545754    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4rv4\" (UniqueName: \"kubernetes.io/projected/d26334f7-7631-47ce-8a86-abd30132fd62-kube-api-access-k4rv4\") pod \"d26334f7-7631-47ce-8a86-abd30132fd62\" (UID: \"d26334f7-7631-47ce-8a86-abd30132fd62\") "
Sep 16 18:08:52.547748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.547718    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26334f7-7631-47ce-8a86-abd30132fd62-kube-api-access-k4rv4" (OuterVolumeSpecName: "kube-api-access-k4rv4") pod "d26334f7-7631-47ce-8a86-abd30132fd62" (UID: "d26334f7-7631-47ce-8a86-abd30132fd62"). InnerVolumeSpecName "kube-api-access-k4rv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:08:52.547748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.547729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d26334f7-7631-47ce-8a86-abd30132fd62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d26334f7-7631-47ce-8a86-abd30132fd62" (UID: "d26334f7-7631-47ce-8a86-abd30132fd62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:08:52.646430 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.646402    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d26334f7-7631-47ce-8a86-abd30132fd62-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:52.646430 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.646427    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rv4\" (UniqueName: \"kubernetes.io/projected/d26334f7-7631-47ce-8a86-abd30132fd62-kube-api-access-k4rv4\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:08:52.683847 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.683821    2560 generic.go:356] "Generic (PLEG): container finished" podID="d26334f7-7631-47ce-8a86-abd30132fd62" containerID="15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1" exitCode=0
Sep 16 18:08:52.683966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.683864    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"
Sep 16 18:08:52.683966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.683895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" event={"ID":"d26334f7-7631-47ce-8a86-abd30132fd62","Type":"ContainerDied","Data":"15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1"}
Sep 16 18:08:52.683966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.683935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" event={"ID":"d26334f7-7631-47ce-8a86-abd30132fd62","Type":"ContainerDied","Data":"b4dd3a77aeb09a97d32c12d22c5b79f049a998941f63f5e2d2485e64c606dbe1"}
Sep 16 18:08:52.683966 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.683956    2560 scope.go:117] "RemoveContainer" containerID="15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1"
Sep 16 18:08:52.692070 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.692027    2560 scope.go:117] "RemoveContainer" containerID="15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1"
Sep 16 18:08:52.692308 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:08:52.692288    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1\": container with ID starting with 15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1 not found: ID does not exist" containerID="15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1"
Sep 16 18:08:52.692374 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.692314    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1"} err="failed to get container status \"15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1\": rpc error: code = NotFound desc = could not find container \"15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1\": container with ID starting with 15514c47fff1b2f7eb1c64f620b5ec427dd3dedec768583ecde9bc6bb6b581b1 not found: ID does not exist"
Sep 16 18:08:52.715986 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.715958    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"]
Sep 16 18:08:52.721098 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:52.721077    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh"]
Sep 16 18:08:53.116678 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:53.116645    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" path="/var/lib/kubelet/pods/d26334f7-7631-47ce-8a86-abd30132fd62/volumes"
Sep 16 18:08:53.192809 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:08:53.192774    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dd5pdh" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.51:9003\" within 1s: context deadline exceeded"
Sep 16 18:09:02.972411 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972374    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"]
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972714    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972727    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972745    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972751    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972760    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="storage-initializer"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972766    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="storage-initializer"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972775    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b23e51-e7cc-4028-baf6-9f36c71d2ebe" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972780    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b23e51-e7cc-4028-baf6-9f36c71d2ebe" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972846    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="20343c9a-c0a4-4800-99a6-7d3d97e65820" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972891    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b23e51-e7cc-4028-baf6-9f36c71d2ebe" containerName="main"
Sep 16 18:09:02.972905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.972900    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d26334f7-7631-47ce-8a86-abd30132fd62" containerName="main"
Sep 16 18:09:02.974980 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.974962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:02.977309 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.977288    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-wwv9d\""
Sep 16 18:09:02.978020 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.977969    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v1-kserve-self-signed-certs\""
Sep 16 18:09:02.978217 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.978196    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Sep 16 18:09:02.978301 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.978226    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Sep 16 18:09:02.991632 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:02.991612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"]
Sep 16 18:09:03.023741 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.023716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.023833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.023745    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-home\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.023833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.023770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgvh6\" (UniqueName: \"kubernetes.io/projected/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-kube-api-access-hgvh6\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.023833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.023796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-dshm\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.023941 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.023842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.023941 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.023861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.124567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.124567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124567    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-home\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.124782 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124610    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgvh6\" (UniqueName: \"kubernetes.io/projected/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-kube-api-access-hgvh6\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.124782 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-dshm\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.124909 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.124909 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.125005 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.124973    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tmp-dir\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.125069 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.125031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-home\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.125193 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.125167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-model-cache\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.126885 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.126866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-dshm\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.127146 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.127130    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tls-certs\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.134792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.134774    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgvh6\" (UniqueName: \"kubernetes.io/projected/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-kube-api-access-hgvh6\") pod \"delw-v1-kserve-6cf8cbb847-bssfw\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.284417 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.284352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:03.570018 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.569935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"]
Sep 16 18:09:03.573490 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:09:03.573455    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ab1bcf_bfcb_49f6_8112_d7bc55147358.slice/crio-b5bcda247763686d1781815e73e87616cfaa503a2938c92e250faf7104002d03 WatchSource:0}: Error finding container b5bcda247763686d1781815e73e87616cfaa503a2938c92e250faf7104002d03: Status 404 returned error can't find the container with id b5bcda247763686d1781815e73e87616cfaa503a2938c92e250faf7104002d03
Sep 16 18:09:03.729968 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.729924    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw" event={"ID":"a9ab1bcf-bfcb-49f6-8112-d7bc55147358","Type":"ContainerStarted","Data":"3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede"}
Sep 16 18:09:03.730206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.729994    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw" event={"ID":"a9ab1bcf-bfcb-49f6-8112-d7bc55147358","Type":"ContainerStarted","Data":"b5bcda247763686d1781815e73e87616cfaa503a2938c92e250faf7104002d03"}
Sep 16 18:09:03.755636 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:03.755565    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw" podStartSLOduration=1.755550162 podStartE2EDuration="1.755550162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:09:03.754930045 +0000 UTC m=+2659.164047293" watchObservedRunningTime="2026-09-16 18:09:03.755550162 +0000 UTC m=+2659.164667410"
Sep 16 18:09:04.284484 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:04.284450    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:09.964019 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:09.963988    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"]
Sep 16 18:09:09.966121 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:09.966103    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:09.968935 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:09.968912    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Sep 16 18:09:09.985135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:09.985112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"]
Sep 16 18:09:10.079272 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.079246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.079416 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.079278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wzl\" (UniqueName: \"kubernetes.io/projected/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-kube-api-access-l8wzl\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.079416 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.079300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.079416 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.079317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.079416 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.079390    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-home\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.079578 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.079470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180161 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180345 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180175    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180345 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-home\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180567 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180542    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180657 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180571    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180706 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180653    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-home\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180755 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180804 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180761    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wzl\" (UniqueName: \"kubernetes.io/projected/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-kube-api-access-l8wzl\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.180860 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.180821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.182397 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.182375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.183012 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.182995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.192445 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.192426    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wzl\" (UniqueName: \"kubernetes.io/projected/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-kube-api-access-l8wzl\") pod \"delw-v2-kserve-5b84cc7ddf-c9n8m\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.277220 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.277163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:10.563924 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.563886    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"]
Sep 16 18:09:10.566451 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:09:10.566421    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f07ec6_1e1a_4675_893d_0ccea6b257dd.slice/crio-577d1e5641109ab74a1ee06d5c80cd1f6aadcf1f21cab67e536d8969fff6ec24 WatchSource:0}: Error finding container 577d1e5641109ab74a1ee06d5c80cd1f6aadcf1f21cab67e536d8969fff6ec24: Status 404 returned error can't find the container with id 577d1e5641109ab74a1ee06d5c80cd1f6aadcf1f21cab67e536d8969fff6ec24
Sep 16 18:09:10.761387 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.761354    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m" event={"ID":"f5f07ec6-1e1a-4675-893d-0ccea6b257dd","Type":"ContainerStarted","Data":"a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d"}
Sep 16 18:09:10.761548 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.761406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m" event={"ID":"f5f07ec6-1e1a-4675-893d-0ccea6b257dd","Type":"ContainerStarted","Data":"577d1e5641109ab74a1ee06d5c80cd1f6aadcf1f21cab67e536d8969fff6ec24"}
Sep 16 18:09:10.789767 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:10.789716    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m" podStartSLOduration=1.789699718 podStartE2EDuration="1.789699718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:09:10.789663414 +0000 UTC m=+2666.198780663" watchObservedRunningTime="2026-09-16 18:09:10.789699718 +0000 UTC m=+2666.198816966"
Sep 16 18:09:11.278216 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:11.278188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:13.284774 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:13.284737    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:13.296992 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:13.296961    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:13.783150 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:13.783117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:09:20.277502 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:20.277460    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:20.289796 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:20.289768    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:20.810910 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:20.810877    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:09:37.467212 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.467137    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"]
Sep 16 18:09:37.469894 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.469877    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.473121 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.473095    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-llp6x\""
Sep 16 18:09:37.473252 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.473135    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 18:09:37.473252 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.473170    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 18:09:37.473252 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.473138    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-s92xv\""
Sep 16 18:09:37.473252 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.473108    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 18:09:37.483928 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.483899    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"]
Sep 16 18:09:37.594562 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.594528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471605cc-815f-482a-b30a-24cfab24ccfe-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.594748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.594575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlk2\" (UniqueName: \"kubernetes.io/projected/471605cc-815f-482a-b30a-24cfab24ccfe-kube-api-access-2mlk2\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.695931 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.695902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471605cc-815f-482a-b30a-24cfab24ccfe-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.696086 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.695945    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlk2\" (UniqueName: \"kubernetes.io/projected/471605cc-815f-482a-b30a-24cfab24ccfe-kube-api-access-2mlk2\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.698329 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.698311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471605cc-815f-482a-b30a-24cfab24ccfe-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.706436 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.706398    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlk2\" (UniqueName: \"kubernetes.io/projected/471605cc-815f-482a-b30a-24cfab24ccfe-kube-api-access-2mlk2\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:37.779989 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:37.779909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:38.052717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:38.052693    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"]
Sep 16 18:09:38.055485 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:09:38.055456    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471605cc_815f_482a_b30a_24cfab24ccfe.slice/crio-60a26d46ba6a2745bf3bf78a0c8cf99e3da9118cc8afe5615fdd80fdd95ab8a9 WatchSource:0}: Error finding container 60a26d46ba6a2745bf3bf78a0c8cf99e3da9118cc8afe5615fdd80fdd95ab8a9: Status 404 returned error can't find the container with id 60a26d46ba6a2745bf3bf78a0c8cf99e3da9118cc8afe5615fdd80fdd95ab8a9
Sep 16 18:09:38.869339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:38.869301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" event={"ID":"471605cc-815f-482a-b30a-24cfab24ccfe","Type":"ContainerStarted","Data":"11296ab5d6fb6101f344ab6018cad8fb1d5364356c6ae48c6663b6dfd4af62ca"}
Sep 16 18:09:38.869826 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:38.869351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" event={"ID":"471605cc-815f-482a-b30a-24cfab24ccfe","Type":"ContainerStarted","Data":"60a26d46ba6a2745bf3bf78a0c8cf99e3da9118cc8afe5615fdd80fdd95ab8a9"}
Sep 16 18:09:38.869826 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:38.869381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:09:38.893149 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:09:38.893107    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podStartSLOduration=1.893093937 podStartE2EDuration="1.893093937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:09:38.890352716 +0000 UTC m=+2694.299469964" watchObservedRunningTime="2026-09-16 18:09:38.893093937 +0000 UTC m=+2694.302211184"
Sep 16 18:10:09.876290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:09.876252    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:10:15.034454 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:15.034417    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"]
Sep 16 18:10:19.876366 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:19.876323    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:10:26.131126 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:26.131088    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"]
Sep 16 18:10:29.876471 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:29.876433    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:10:30.053404 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:30.053364    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main" containerID="cri-o://11296ab5d6fb6101f344ab6018cad8fb1d5364356c6ae48c6663b6dfd4af62ca" gracePeriod=45
Sep 16 18:10:39.877066 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:39.877025    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:10:39.878582 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:39.877157    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:10:41.149658 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.149615    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m" podUID="f5f07ec6-1e1a-4675-893d-0ccea6b257dd" containerName="main" containerID="cri-o://a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d" gracePeriod=45
Sep 16 18:10:41.398073 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.398042    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:10:41.517409 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517374    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tls-certs\") pod \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") "
Sep 16 18:10:41.517409 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517419    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tmp-dir\") pod \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") "
Sep 16 18:10:41.517717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517440    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-dshm\") pod \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") "
Sep 16 18:10:41.517717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517463    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-home\") pod \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") "
Sep 16 18:10:41.517717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517515    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8wzl\" (UniqueName: \"kubernetes.io/projected/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-kube-api-access-l8wzl\") pod \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") "
Sep 16 18:10:41.517717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517535    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-model-cache\") pod \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\" (UID: \"f5f07ec6-1e1a-4675-893d-0ccea6b257dd\") "
Sep 16 18:10:41.517925 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5f07ec6-1e1a-4675-893d-0ccea6b257dd" (UID: "f5f07ec6-1e1a-4675-893d-0ccea6b257dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:41.517925 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-home" (OuterVolumeSpecName: "home") pod "f5f07ec6-1e1a-4675-893d-0ccea6b257dd" (UID: "f5f07ec6-1e1a-4675-893d-0ccea6b257dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:41.517995 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.517918    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5f07ec6-1e1a-4675-893d-0ccea6b257dd" (UID: "f5f07ec6-1e1a-4675-893d-0ccea6b257dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:41.519776 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.519746    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-dshm" (OuterVolumeSpecName: "dshm") pod "f5f07ec6-1e1a-4675-893d-0ccea6b257dd" (UID: "f5f07ec6-1e1a-4675-893d-0ccea6b257dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:41.519908 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.519845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-kube-api-access-l8wzl" (OuterVolumeSpecName: "kube-api-access-l8wzl") pod "f5f07ec6-1e1a-4675-893d-0ccea6b257dd" (UID: "f5f07ec6-1e1a-4675-893d-0ccea6b257dd"). InnerVolumeSpecName "kube-api-access-l8wzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:10:41.519908 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.519859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5f07ec6-1e1a-4675-893d-0ccea6b257dd" (UID: "f5f07ec6-1e1a-4675-893d-0ccea6b257dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:10:41.618610 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.618537    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:41.618610 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.618575    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:41.618610 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.618585    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:41.618610 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.618616    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wzl\" (UniqueName: \"kubernetes.io/projected/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-kube-api-access-l8wzl\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:41.618610 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.618625    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:41.618911 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:41.618633    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5f07ec6-1e1a-4675-893d-0ccea6b257dd-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:42.125546 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.125507    2560 generic.go:356] "Generic (PLEG): container finished" podID="f5f07ec6-1e1a-4675-893d-0ccea6b257dd" containerID="a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d" exitCode=0
Sep 16 18:10:42.125776 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.125559    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"
Sep 16 18:10:42.125776 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.125610    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m" event={"ID":"f5f07ec6-1e1a-4675-893d-0ccea6b257dd","Type":"ContainerDied","Data":"a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d"}
Sep 16 18:10:42.125776 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.125664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m" event={"ID":"f5f07ec6-1e1a-4675-893d-0ccea6b257dd","Type":"ContainerDied","Data":"577d1e5641109ab74a1ee06d5c80cd1f6aadcf1f21cab67e536d8969fff6ec24"}
Sep 16 18:10:42.125776 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.125690    2560 scope.go:117] "RemoveContainer" containerID="a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d"
Sep 16 18:10:42.139760 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.139734    2560 scope.go:117] "RemoveContainer" containerID="a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d"
Sep 16 18:10:42.140138 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:10:42.140077    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d\": container with ID starting with a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d not found: ID does not exist" containerID="a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d"
Sep 16 18:10:42.140138 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.140106    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d"} err="failed to get container status \"a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d\": rpc error: code = NotFound desc = could not find container \"a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d\": container with ID starting with a6b4f3bcd0fc30afc78112bd9aa0547e46563d8b015978deccf583c50a8af02d not found: ID does not exist"
Sep 16 18:10:42.157365 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.157337    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"]
Sep 16 18:10:42.164400 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.164372    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-c9n8m"]
Sep 16 18:10:42.399721 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:42.399638    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"]
Sep 16 18:10:43.117179 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:43.117136    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5f07ec6-1e1a-4675-893d-0ccea6b257dd" path="/var/lib/kubelet/pods/f5f07ec6-1e1a-4675-893d-0ccea6b257dd/volumes"
Sep 16 18:10:49.876776 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:49.876731    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:10:57.418150 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.418104    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw" podUID="a9ab1bcf-bfcb-49f6-8112-d7bc55147358" containerName="main" containerID="cri-o://3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede" gracePeriod=45
Sep 16 18:10:57.654191 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.654165    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:10:57.751403 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751372    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-model-cache\") pod \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") "
Sep 16 18:10:57.751557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751451    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-home\") pod \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") "
Sep 16 18:10:57.751557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751478    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tls-certs\") pod \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") "
Sep 16 18:10:57.751557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751530    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgvh6\" (UniqueName: \"kubernetes.io/projected/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-kube-api-access-hgvh6\") pod \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") "
Sep 16 18:10:57.751557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751553    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-dshm\") pod \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") "
Sep 16 18:10:57.751811 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751662    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-model-cache" (OuterVolumeSpecName: "model-cache") pod "a9ab1bcf-bfcb-49f6-8112-d7bc55147358" (UID: "a9ab1bcf-bfcb-49f6-8112-d7bc55147358"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:57.751811 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751718    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tmp-dir\") pod \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\" (UID: \"a9ab1bcf-bfcb-49f6-8112-d7bc55147358\") "
Sep 16 18:10:57.751916 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751726    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-home" (OuterVolumeSpecName: "home") pod "a9ab1bcf-bfcb-49f6-8112-d7bc55147358" (UID: "a9ab1bcf-bfcb-49f6-8112-d7bc55147358"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:57.751916 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a9ab1bcf-bfcb-49f6-8112-d7bc55147358" (UID: "a9ab1bcf-bfcb-49f6-8112-d7bc55147358"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:57.752037 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.751989    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:57.752037 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.752007    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:57.752037 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.752018    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:57.753685 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.753662    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-dshm" (OuterVolumeSpecName: "dshm") pod "a9ab1bcf-bfcb-49f6-8112-d7bc55147358" (UID: "a9ab1bcf-bfcb-49f6-8112-d7bc55147358"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:10:57.753786 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.753700    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-kube-api-access-hgvh6" (OuterVolumeSpecName: "kube-api-access-hgvh6") pod "a9ab1bcf-bfcb-49f6-8112-d7bc55147358" (UID: "a9ab1bcf-bfcb-49f6-8112-d7bc55147358"). InnerVolumeSpecName "kube-api-access-hgvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:10:57.753786 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.753714    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a9ab1bcf-bfcb-49f6-8112-d7bc55147358" (UID: "a9ab1bcf-bfcb-49f6-8112-d7bc55147358"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:10:57.853290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.853256    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:57.853290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.853286    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgvh6\" (UniqueName: \"kubernetes.io/projected/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-kube-api-access-hgvh6\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:57.853290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:57.853296    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9ab1bcf-bfcb-49f6-8112-d7bc55147358-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:10:58.190192 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.190101    2560 generic.go:356] "Generic (PLEG): container finished" podID="a9ab1bcf-bfcb-49f6-8112-d7bc55147358" containerID="3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede" exitCode=0
Sep 16 18:10:58.190192 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.190145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"
Sep 16 18:10:58.190425 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.190147    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw" event={"ID":"a9ab1bcf-bfcb-49f6-8112-d7bc55147358","Type":"ContainerDied","Data":"3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede"}
Sep 16 18:10:58.190425 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.190304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw" event={"ID":"a9ab1bcf-bfcb-49f6-8112-d7bc55147358","Type":"ContainerDied","Data":"b5bcda247763686d1781815e73e87616cfaa503a2938c92e250faf7104002d03"}
Sep 16 18:10:58.190425 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.190339    2560 scope.go:117] "RemoveContainer" containerID="3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede"
Sep 16 18:10:58.198774 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.198755    2560 scope.go:117] "RemoveContainer" containerID="3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede"
Sep 16 18:10:58.199029 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:10:58.199013    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede\": container with ID starting with 3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede not found: ID does not exist" containerID="3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede"
Sep 16 18:10:58.199080 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.199036    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede"} err="failed to get container status \"3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede\": rpc error: code = NotFound desc = could not find container \"3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede\": container with ID starting with 3567d0cc5e6a65aaa7d582d7e27448c6f6ad5e208c47d0e33126798a43482ede not found: ID does not exist"
Sep 16 18:10:58.221754 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.221708    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"]
Sep 16 18:10:58.231390 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:58.231368    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v1-kserve-6cf8cbb847-bssfw"]
Sep 16 18:10:59.117003 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:59.116970    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9ab1bcf-bfcb-49f6-8112-d7bc55147358" path="/var/lib/kubelet/pods/a9ab1bcf-bfcb-49f6-8112-d7bc55147358/volumes"
Sep 16 18:10:59.876285 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:10:59.876242    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:11:00.201668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.201633    2560 generic.go:356] "Generic (PLEG): container finished" podID="471605cc-815f-482a-b30a-24cfab24ccfe" containerID="11296ab5d6fb6101f344ab6018cad8fb1d5364356c6ae48c6663b6dfd4af62ca" exitCode=0
Sep 16 18:11:00.202032 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.201706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" event={"ID":"471605cc-815f-482a-b30a-24cfab24ccfe","Type":"ContainerDied","Data":"11296ab5d6fb6101f344ab6018cad8fb1d5364356c6ae48c6663b6dfd4af62ca"}
Sep 16 18:11:00.295198 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.295167    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:11:00.371962 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.371923    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mlk2\" (UniqueName: \"kubernetes.io/projected/471605cc-815f-482a-b30a-24cfab24ccfe-kube-api-access-2mlk2\") pod \"471605cc-815f-482a-b30a-24cfab24ccfe\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") "
Sep 16 18:11:00.372133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.371981    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471605cc-815f-482a-b30a-24cfab24ccfe-tls-certs\") pod \"471605cc-815f-482a-b30a-24cfab24ccfe\" (UID: \"471605cc-815f-482a-b30a-24cfab24ccfe\") "
Sep 16 18:11:00.374267 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.374239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/471605cc-815f-482a-b30a-24cfab24ccfe-kube-api-access-2mlk2" (OuterVolumeSpecName: "kube-api-access-2mlk2") pod "471605cc-815f-482a-b30a-24cfab24ccfe" (UID: "471605cc-815f-482a-b30a-24cfab24ccfe"). InnerVolumeSpecName "kube-api-access-2mlk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:11:00.374357 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.374249    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/471605cc-815f-482a-b30a-24cfab24ccfe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "471605cc-815f-482a-b30a-24cfab24ccfe" (UID: "471605cc-815f-482a-b30a-24cfab24ccfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:11:00.473275 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.473248    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/471605cc-815f-482a-b30a-24cfab24ccfe-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:11:00.473275 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:00.473272    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlk2\" (UniqueName: \"kubernetes.io/projected/471605cc-815f-482a-b30a-24cfab24ccfe-kube-api-access-2mlk2\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:11:01.207069 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:01.207036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"
Sep 16 18:11:01.207476 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:01.207068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7" event={"ID":"471605cc-815f-482a-b30a-24cfab24ccfe","Type":"ContainerDied","Data":"60a26d46ba6a2745bf3bf78a0c8cf99e3da9118cc8afe5615fdd80fdd95ab8a9"}
Sep 16 18:11:01.207476 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:01.207115    2560 scope.go:117] "RemoveContainer" containerID="11296ab5d6fb6101f344ab6018cad8fb1d5364356c6ae48c6663b6dfd4af62ca"
Sep 16 18:11:01.224770 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:01.224709    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"]
Sep 16 18:11:01.228194 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:01.228173    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schec46b7"]
Sep 16 18:11:03.117736 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:03.117697    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" path="/var/lib/kubelet/pods/471605cc-815f-482a-b30a-24cfab24ccfe/volumes"
Sep 16 18:11:11.664953 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.664878    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"]
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665156    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665168    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665176    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9ab1bcf-bfcb-49f6-8112-d7bc55147358" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665182    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9ab1bcf-bfcb-49f6-8112-d7bc55147358" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665192    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5f07ec6-1e1a-4675-893d-0ccea6b257dd" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665198    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5f07ec6-1e1a-4675-893d-0ccea6b257dd" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665249    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5f07ec6-1e1a-4675-893d-0ccea6b257dd" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665256    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9ab1bcf-bfcb-49f6-8112-d7bc55147358" containerName="main"
Sep 16 18:11:11.665317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.665265    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="471605cc-815f-482a-b30a-24cfab24ccfe" containerName="main"
Sep 16 18:11:11.669433 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.669411    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.672712 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.672688    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 16 18:11:11.672813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.672693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 16 18:11:11.673434 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.673415    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-9qcfc\""
Sep 16 18:11:11.677412 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.677394    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 16 18:11:11.682041 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.682017    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"]
Sep 16 18:11:11.758258 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.758219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.758436 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.758283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-dshm\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.758436 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.758331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea13b954-c66c-42b5-a4b2-b89b8171c242-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.758436 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.758405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-home\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.758436 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.758437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.758632 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.758455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwgnp\" (UniqueName: \"kubernetes.io/projected/ea13b954-c66c-42b5-a4b2-b89b8171c242-kube-api-access-dwgnp\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859390 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwgnp\" (UniqueName: \"kubernetes.io/projected/ea13b954-c66c-42b5-a4b2-b89b8171c242-kube-api-access-dwgnp\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859579 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859579 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-dshm\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859579 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea13b954-c66c-42b5-a4b2-b89b8171c242-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859579 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-home\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859579 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859556    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859900 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859959 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.859999 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.859958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-home\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.861833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.861813    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-dshm\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.862031 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.862012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea13b954-c66c-42b5-a4b2-b89b8171c242-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.867238 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.867214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwgnp\" (UniqueName: \"kubernetes.io/projected/ea13b954-c66c-42b5-a4b2-b89b8171c242-kube-api-access-dwgnp\") pod \"roll-v2-kserve-d4dbf9d67-tbqzp\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:11.979815 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:11.979794    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:12.257019 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:12.256938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"]
Sep 16 18:11:12.260069 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:11:12.260042    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea13b954_c66c_42b5_a4b2_b89b8171c242.slice/crio-f62f1c1032dd91088c8b13178b816af8e9cba3618217f1c5d37274fde504c3d9 WatchSource:0}: Error finding container f62f1c1032dd91088c8b13178b816af8e9cba3618217f1c5d37274fde504c3d9: Status 404 returned error can't find the container with id f62f1c1032dd91088c8b13178b816af8e9cba3618217f1c5d37274fde504c3d9
Sep 16 18:11:13.256483 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:13.256446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" event={"ID":"ea13b954-c66c-42b5-a4b2-b89b8171c242","Type":"ContainerStarted","Data":"d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff"}
Sep 16 18:11:13.256890 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:13.256508    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" event={"ID":"ea13b954-c66c-42b5-a4b2-b89b8171c242","Type":"ContainerStarted","Data":"f62f1c1032dd91088c8b13178b816af8e9cba3618217f1c5d37274fde504c3d9"}
Sep 16 18:11:13.281017 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:13.280956    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" podStartSLOduration=2.280936844 podStartE2EDuration="2.280936844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:11:13.280736487 +0000 UTC m=+2788.689853735" watchObservedRunningTime="2026-09-16 18:11:13.280936844 +0000 UTC m=+2788.690054093"
Sep 16 18:11:13.980283 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:13.980252    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:21.980152 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:21.980114    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:21.992395 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:21.992374    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:22.304791 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:22.304708    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:11:45.203159 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.203127    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"]
Sep 16 18:11:45.206565 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.206545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.215994 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.215975    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 18:11:45.216276 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.216254    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9t6sk\""
Sep 16 18:11:45.216364 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.216295    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-6mf7d\""
Sep 16 18:11:45.216364 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.216348    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 18:11:45.217239 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.217222    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 18:11:45.227494 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.227470    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"]
Sep 16 18:11:45.332160 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.332114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.332160 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.332162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9htv\" (UniqueName: \"kubernetes.io/projected/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-kube-api-access-g9htv\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.433171 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.433142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.433269 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.433185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9htv\" (UniqueName: \"kubernetes.io/projected/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-kube-api-access-g9htv\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.435679 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.435654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.441201 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.441175    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9htv\" (UniqueName: \"kubernetes.io/projected/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-kube-api-access-g9htv\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.516503 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.516477    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:45.790662 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:45.790637    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"]
Sep 16 18:11:45.792977 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:11:45.792937    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b39fe8d_4ae5_4f0a_990b_495c5eb3847f.slice/crio-47d578ab0987422dff602305d9cb8f9b2a2ae1bf868c582589ce17e8ca101af1 WatchSource:0}: Error finding container 47d578ab0987422dff602305d9cb8f9b2a2ae1bf868c582589ce17e8ca101af1: Status 404 returned error can't find the container with id 47d578ab0987422dff602305d9cb8f9b2a2ae1bf868c582589ce17e8ca101af1
Sep 16 18:11:46.387565 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:46.387527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" event={"ID":"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f","Type":"ContainerStarted","Data":"fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a"}
Sep 16 18:11:46.388024 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:46.387582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" event={"ID":"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f","Type":"ContainerStarted","Data":"47d578ab0987422dff602305d9cb8f9b2a2ae1bf868c582589ce17e8ca101af1"}
Sep 16 18:11:46.388024 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:46.387627    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:11:46.419655 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:11:46.419607    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" podStartSLOduration=1.4195801989999999 podStartE2EDuration="1.419580199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:11:46.417408422 +0000 UTC m=+2821.826525674" watchObservedRunningTime="2026-09-16 18:11:46.419580199 +0000 UTC m=+2821.828697447"
Sep 16 18:12:17.393919 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:12:17.393888    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:13:41.149027 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:41.148993    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"]
Sep 16 18:13:56.168264 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.168221    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" podUID="ea13b954-c66c-42b5-a4b2-b89b8171c242" containerName="main" containerID="cri-o://d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff" gracePeriod=45
Sep 16 18:13:56.294488 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.294448    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" podUID="ea13b954-c66c-42b5-a4b2-b89b8171c242" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 16 18:13:56.411078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.411056    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:13:56.474102 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474061    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-model-cache\") pod \"ea13b954-c66c-42b5-a4b2-b89b8171c242\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") "
Sep 16 18:13:56.474280 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-tmp-dir\") pod \"ea13b954-c66c-42b5-a4b2-b89b8171c242\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") "
Sep 16 18:13:56.474280 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474143    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-dshm\") pod \"ea13b954-c66c-42b5-a4b2-b89b8171c242\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") "
Sep 16 18:13:56.474280 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474159    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea13b954-c66c-42b5-a4b2-b89b8171c242-tls-certs\") pod \"ea13b954-c66c-42b5-a4b2-b89b8171c242\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") "
Sep 16 18:13:56.474280 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-home\") pod \"ea13b954-c66c-42b5-a4b2-b89b8171c242\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") "
Sep 16 18:13:56.474280 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwgnp\" (UniqueName: \"kubernetes.io/projected/ea13b954-c66c-42b5-a4b2-b89b8171c242-kube-api-access-dwgnp\") pod \"ea13b954-c66c-42b5-a4b2-b89b8171c242\" (UID: \"ea13b954-c66c-42b5-a4b2-b89b8171c242\") "
Sep 16 18:13:56.474517 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474401    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea13b954-c66c-42b5-a4b2-b89b8171c242" (UID: "ea13b954-c66c-42b5-a4b2-b89b8171c242"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:56.474517 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea13b954-c66c-42b5-a4b2-b89b8171c242" (UID: "ea13b954-c66c-42b5-a4b2-b89b8171c242"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:56.474517 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.474458    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-home" (OuterVolumeSpecName: "home") pod "ea13b954-c66c-42b5-a4b2-b89b8171c242" (UID: "ea13b954-c66c-42b5-a4b2-b89b8171c242"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:56.476342 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.476314    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-dshm" (OuterVolumeSpecName: "dshm") pod "ea13b954-c66c-42b5-a4b2-b89b8171c242" (UID: "ea13b954-c66c-42b5-a4b2-b89b8171c242"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:13:56.476444 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.476366    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea13b954-c66c-42b5-a4b2-b89b8171c242-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea13b954-c66c-42b5-a4b2-b89b8171c242" (UID: "ea13b954-c66c-42b5-a4b2-b89b8171c242"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:13:56.476444 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.476389    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea13b954-c66c-42b5-a4b2-b89b8171c242-kube-api-access-dwgnp" (OuterVolumeSpecName: "kube-api-access-dwgnp") pod "ea13b954-c66c-42b5-a4b2-b89b8171c242" (UID: "ea13b954-c66c-42b5-a4b2-b89b8171c242"). InnerVolumeSpecName "kube-api-access-dwgnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:13:56.575462 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.575435    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:13:56.575462 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.575460    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwgnp\" (UniqueName: \"kubernetes.io/projected/ea13b954-c66c-42b5-a4b2-b89b8171c242-kube-api-access-dwgnp\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:13:56.575668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.575471    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:13:56.575668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.575480    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:13:56.575668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.575488    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea13b954-c66c-42b5-a4b2-b89b8171c242-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:13:56.575668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.575496    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea13b954-c66c-42b5-a4b2-b89b8171c242-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:13:56.902968 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.902878    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea13b954-c66c-42b5-a4b2-b89b8171c242" containerID="d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff" exitCode=0
Sep 16 18:13:56.902968 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.902920    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" event={"ID":"ea13b954-c66c-42b5-a4b2-b89b8171c242","Type":"ContainerDied","Data":"d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff"}
Sep 16 18:13:56.902968 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.902941    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"
Sep 16 18:13:56.902968 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.902966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp" event={"ID":"ea13b954-c66c-42b5-a4b2-b89b8171c242","Type":"ContainerDied","Data":"f62f1c1032dd91088c8b13178b816af8e9cba3618217f1c5d37274fde504c3d9"}
Sep 16 18:13:56.903251 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.902988    2560 scope.go:117] "RemoveContainer" containerID="d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff"
Sep 16 18:13:56.913940 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.913835    2560 scope.go:117] "RemoveContainer" containerID="d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff"
Sep 16 18:13:56.914211 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:13:56.914185    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff\": container with ID starting with d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff not found: ID does not exist" containerID="d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff"
Sep 16 18:13:56.914315 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.914222    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff"} err="failed to get container status \"d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff\": rpc error: code = NotFound desc = could not find container \"d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff\": container with ID starting with d71f1ec49f50e87e2c08094c42e2cf9c817ddbaa1877ba840b339a0053293dff not found: ID does not exist"
Sep 16 18:13:56.933324 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.933302    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"]
Sep 16 18:13:56.942049 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:56.942024    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tbqzp"]
Sep 16 18:13:57.117145 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:13:57.117111    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea13b954-c66c-42b5-a4b2-b89b8171c242" path="/var/lib/kubelet/pods/ea13b954-c66c-42b5-a4b2-b89b8171c242/volumes"
Sep 16 18:14:21.210040 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:21.209943    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"]
Sep 16 18:14:27.393936 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:27.393895    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:14:36.236290 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:36.236253    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main" containerID="cri-o://fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a" gracePeriod=45
Sep 16 18:14:37.393529 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:37.393494    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:14:47.393838 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:47.393801    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:14:47.394207 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:47.393902    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:14:57.393759 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:14:57.393724    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:15:06.476813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.476780    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:15:06.608671 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.608572    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9htv\" (UniqueName: \"kubernetes.io/projected/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-kube-api-access-g9htv\") pod \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") "
Sep 16 18:15:06.608671 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.608666    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-tls-certs\") pod \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\" (UID: \"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f\") "
Sep 16 18:15:06.610702 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.610661    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" (UID: "0b39fe8d-4ae5-4f0a-990b-495c5eb3847f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:15:06.610702 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.610691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-kube-api-access-g9htv" (OuterVolumeSpecName: "kube-api-access-g9htv") pod "0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" (UID: "0b39fe8d-4ae5-4f0a-990b-495c5eb3847f"). InnerVolumeSpecName "kube-api-access-g9htv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:15:06.709718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.709687    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:15:06.709718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:06.709713    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9htv\" (UniqueName: \"kubernetes.io/projected/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f-kube-api-access-g9htv\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:15:07.173194 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.173163    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerID="fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a" exitCode=0
Sep 16 18:15:07.173339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.173209    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"
Sep 16 18:15:07.173339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.173227    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" event={"ID":"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f","Type":"ContainerDied","Data":"fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a"}
Sep 16 18:15:07.173339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.173269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg" event={"ID":"0b39fe8d-4ae5-4f0a-990b-495c5eb3847f","Type":"ContainerDied","Data":"47d578ab0987422dff602305d9cb8f9b2a2ae1bf868c582589ce17e8ca101af1"}
Sep 16 18:15:07.173339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.173290    2560 scope.go:117] "RemoveContainer" containerID="fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a"
Sep 16 18:15:07.181266 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.181248    2560 scope.go:117] "RemoveContainer" containerID="fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a"
Sep 16 18:15:07.181530 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:15:07.181509    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a\": container with ID starting with fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a not found: ID does not exist" containerID="fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a"
Sep 16 18:15:07.181622 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.181543    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a"} err="failed to get container status \"fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a\": rpc error: code = NotFound desc = could not find container \"fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a\": container with ID starting with fdc5cc413d358b08aff37278cfdb06694ec99e04976304490933a80b11838d8a not found: ID does not exist"
Sep 16 18:15:07.200522 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.200498    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"]
Sep 16 18:15:07.207030 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:07.207010    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schejlmlg"]
Sep 16 18:15:09.117637 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:09.117606    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" path="/var/lib/kubelet/pods/0b39fe8d-4ae5-4f0a-990b-495c5eb3847f/volumes"
Sep 16 18:15:15.734558 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734521    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"]
Sep 16 18:15:15.734945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734836    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main"
Sep 16 18:15:15.734945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734848    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main"
Sep 16 18:15:15.734945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734859    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea13b954-c66c-42b5-a4b2-b89b8171c242" containerName="main"
Sep 16 18:15:15.734945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734864    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea13b954-c66c-42b5-a4b2-b89b8171c242" containerName="main"
Sep 16 18:15:15.734945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734908    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b39fe8d-4ae5-4f0a-990b-495c5eb3847f" containerName="main"
Sep 16 18:15:15.734945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.734918    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea13b954-c66c-42b5-a4b2-b89b8171c242" containerName="main"
Sep 16 18:15:15.739241 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.739218    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:15.743099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.743004    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 18:15:15.743228 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.743179    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-nn4ds\""
Sep 16 18:15:15.743301 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.743219    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 18:15:15.744023 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.743630    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-9bp9x\""
Sep 16 18:15:15.744023 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.743699    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 18:15:15.752330 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.752304    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"]
Sep 16 18:15:15.889702 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.889668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/6326a807-74ad-461e-8b15-d609ab421816-kube-api-access-d6tsm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:15.889702 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.889712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6326a807-74ad-461e-8b15-d609ab421816-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:15.990974 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.990883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/6326a807-74ad-461e-8b15-d609ab421816-kube-api-access-d6tsm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:15.990974 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.990925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6326a807-74ad-461e-8b15-d609ab421816-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:15.993374 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:15.993352    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6326a807-74ad-461e-8b15-d609ab421816-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:16.005057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:16.005028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/6326a807-74ad-461e-8b15-d609ab421816-kube-api-access-d6tsm\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:16.055121 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:16.055094    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:16.326241 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:16.326168    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"]
Sep 16 18:15:16.329226 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:15:16.329201    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6326a807_74ad_461e_8b15_d609ab421816.slice/crio-a2d6d5de45d712835eaf1878732c6d85db0439318607bc6d7d581e44fdceb5d8 WatchSource:0}: Error finding container a2d6d5de45d712835eaf1878732c6d85db0439318607bc6d7d581e44fdceb5d8: Status 404 returned error can't find the container with id a2d6d5de45d712835eaf1878732c6d85db0439318607bc6d7d581e44fdceb5d8
Sep 16 18:15:17.215025 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:17.214989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" event={"ID":"6326a807-74ad-461e-8b15-d609ab421816","Type":"ContainerStarted","Data":"388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e"}
Sep 16 18:15:17.215410 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:17.215040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" event={"ID":"6326a807-74ad-461e-8b15-d609ab421816","Type":"ContainerStarted","Data":"a2d6d5de45d712835eaf1878732c6d85db0439318607bc6d7d581e44fdceb5d8"}
Sep 16 18:15:17.215410 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:17.215130    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:15:17.239705 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:17.239646    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podStartSLOduration=2.239630899 podStartE2EDuration="2.239630899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:15:17.239273227 +0000 UTC m=+3032.648390474" watchObservedRunningTime="2026-09-16 18:15:17.239630899 +0000 UTC m=+3032.648748147"
Sep 16 18:15:48.220480 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:15:48.220447    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:16:41.137626 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.137581    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"]
Sep 16 18:16:41.140977 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.140956    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.143116 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.143096    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 18:16:41.151286 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.151262    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"]
Sep 16 18:16:41.262698 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.262662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v848\" (UniqueName: \"kubernetes.io/projected/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-kube-api-access-2v848\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.262884 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.262715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.262884 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.262783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.262884 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.262829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.263045 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.262887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.263045 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.262993    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.363780 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.363743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.363965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.363799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v848\" (UniqueName: \"kubernetes.io/projected/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-kube-api-access-2v848\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.363965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.363821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.363965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.363845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.363965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.363861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.363965 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.363881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.364288 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.364214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.364354 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.364312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.364412 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.364369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.366132 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.366113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.366434 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.366413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.376158 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.376140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v848\" (UniqueName: \"kubernetes.io/projected/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-kube-api-access-2v848\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.451903 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.451814    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:41.728534 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:41.728467    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"]
Sep 16 18:16:41.731227 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:16:41.731199    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0fe281_7fe8_45a7_9f1a_e2c26fb98642.slice/crio-c5fccc663d82af033ea6003bbc5823abfbc3a3ea4c47872a59a9aa374a8bbe12 WatchSource:0}: Error finding container c5fccc663d82af033ea6003bbc5823abfbc3a3ea4c47872a59a9aa374a8bbe12: Status 404 returned error can't find the container with id c5fccc663d82af033ea6003bbc5823abfbc3a3ea4c47872a59a9aa374a8bbe12
Sep 16 18:16:42.548668 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:42.548632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" event={"ID":"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642","Type":"ContainerStarted","Data":"1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"}
Sep 16 18:16:42.549066 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:42.548692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" event={"ID":"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642","Type":"ContainerStarted","Data":"c5fccc663d82af033ea6003bbc5823abfbc3a3ea4c47872a59a9aa374a8bbe12"}
Sep 16 18:16:42.567101 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:42.567051    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" podStartSLOduration=1.567034478 podStartE2EDuration="1.567034478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:16:42.56560837 +0000 UTC m=+3117.974725612" watchObservedRunningTime="2026-09-16 18:16:42.567034478 +0000 UTC m=+3117.976151725"
Sep 16 18:16:43.452928 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:43.452885    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:51.452266 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:51.452231    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:51.465077 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:51.465052    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:16:51.596836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:16:51.596801    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:17:14.903937 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:14.903853    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"]
Sep 16 18:17:14.913346 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:17:14.913313    2560 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:17:14.913346 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:17:14.913346 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:17:14.913346 ip-10-0-128-156 kubenswrapper[2560]:  > containerID="1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362" cmd=["/bin/sleep","15"]
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:14.913362    2560 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" message="" err=<
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]:  >
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:17:14.913375    2560 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]:         , stdout: , stderr: , exit code -1
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" podUID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" containerName="main" containerID="cri-o://1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"
Sep 16 18:17:14.913601 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:14.913406    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" podUID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" containerName="main" containerID="cri-o://1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362" gracePeriod=60
Sep 16 18:17:15.275862 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.275827    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:17:15.330088 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330040    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-dshm\") pod \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") "
Sep 16 18:17:15.330088 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330078    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tmp-dir\") pod \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") "
Sep 16 18:17:15.330328 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330119    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tls-certs\") pod \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") "
Sep 16 18:17:15.330328 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330176    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-model-cache\") pod \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") "
Sep 16 18:17:15.330328 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-home\") pod \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") "
Sep 16 18:17:15.330328 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330238    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v848\" (UniqueName: \"kubernetes.io/projected/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-kube-api-access-2v848\") pod \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\" (UID: \"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642\") "
Sep 16 18:17:15.330547 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330399    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" (UID: "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:17:15.330547 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330415    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" (UID: "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:17:15.330547 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330478    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:17:15.330547 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330495    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:17:15.330792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.330682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-home" (OuterVolumeSpecName: "home") pod "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" (UID: "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:17:15.332403 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.332378    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-dshm" (OuterVolumeSpecName: "dshm") pod "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" (UID: "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:17:15.332539 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.332442    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" (UID: "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:17:15.332539 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.332518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-kube-api-access-2v848" (OuterVolumeSpecName: "kube-api-access-2v848") pod "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" (UID: "8b0fe281-7fe8-45a7-9f1a-e2c26fb98642"). InnerVolumeSpecName "kube-api-access-2v848". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:17:15.431881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.431846    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v848\" (UniqueName: \"kubernetes.io/projected/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-kube-api-access-2v848\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:17:15.431881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.431876    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:17:15.431881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.431886    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:17:15.431881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.431894    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:17:15.684970 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.684935    2560 generic.go:356] "Generic (PLEG): container finished" podID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" containerID="1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362" exitCode=0
Sep 16 18:17:15.685132 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.684978    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" event={"ID":"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642","Type":"ContainerDied","Data":"1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"}
Sep 16 18:17:15.685132 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.684985    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"
Sep 16 18:17:15.685132 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.685028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf" event={"ID":"8b0fe281-7fe8-45a7-9f1a-e2c26fb98642","Type":"ContainerDied","Data":"c5fccc663d82af033ea6003bbc5823abfbc3a3ea4c47872a59a9aa374a8bbe12"}
Sep 16 18:17:15.685132 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.685053    2560 scope.go:117] "RemoveContainer" containerID="1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"
Sep 16 18:17:15.693915 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.693896    2560 scope.go:117] "RemoveContainer" containerID="1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"
Sep 16 18:17:15.694200 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:17:15.694178    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362\": container with ID starting with 1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362 not found: ID does not exist" containerID="1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"
Sep 16 18:17:15.694279 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.694206    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362"} err="failed to get container status \"1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362\": rpc error: code = NotFound desc = could not find container \"1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362\": container with ID starting with 1ec59bd98d74784d503df1269599a048039b6a8b660323a4937c4d124df0e362 not found: ID does not exist"
Sep 16 18:17:15.704972 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.704951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"]
Sep 16 18:17:15.709246 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:15.709224    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-bbqwf"]
Sep 16 18:17:17.117826 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:17:17.117784    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" path="/var/lib/kubelet/pods/8b0fe281-7fe8-45a7-9f1a-e2c26fb98642/volumes"
Sep 16 18:18:04.974007 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:04.973961    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"]
Sep 16 18:18:08.221288 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:08.221245    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:18:18.220745 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:18.220706    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:18:19.992278 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:19.992233    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main" containerID="cri-o://388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e" gracePeriod=45
Sep 16 18:18:28.220670 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:28.220631    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:18:28.221046 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:28.220732    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:18:38.221055 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:38.220965    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:18:48.220925 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:48.220877    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:18:50.221867 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.221843    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:18:50.382842 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.382734    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/6326a807-74ad-461e-8b15-d609ab421816-kube-api-access-d6tsm\") pod \"6326a807-74ad-461e-8b15-d609ab421816\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") "
Sep 16 18:18:50.382842 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.382829    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6326a807-74ad-461e-8b15-d609ab421816-tls-certs\") pod \"6326a807-74ad-461e-8b15-d609ab421816\" (UID: \"6326a807-74ad-461e-8b15-d609ab421816\") "
Sep 16 18:18:50.384973 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.384943    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6326a807-74ad-461e-8b15-d609ab421816-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6326a807-74ad-461e-8b15-d609ab421816" (UID: "6326a807-74ad-461e-8b15-d609ab421816"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:18:50.385089 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.384988    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6326a807-74ad-461e-8b15-d609ab421816-kube-api-access-d6tsm" (OuterVolumeSpecName: "kube-api-access-d6tsm") pod "6326a807-74ad-461e-8b15-d609ab421816" (UID: "6326a807-74ad-461e-8b15-d609ab421816"). InnerVolumeSpecName "kube-api-access-d6tsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:18:50.483940 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.483897    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6tsm\" (UniqueName: \"kubernetes.io/projected/6326a807-74ad-461e-8b15-d609ab421816-kube-api-access-d6tsm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:18:50.483940 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:50.483936    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6326a807-74ad-461e-8b15-d609ab421816-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:18:51.061130 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.061095    2560 generic.go:356] "Generic (PLEG): container finished" podID="6326a807-74ad-461e-8b15-d609ab421816" containerID="388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e" exitCode=0
Sep 16 18:18:51.061307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.061144    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"
Sep 16 18:18:51.061307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.061163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" event={"ID":"6326a807-74ad-461e-8b15-d609ab421816","Type":"ContainerDied","Data":"388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e"}
Sep 16 18:18:51.061307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.061250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht" event={"ID":"6326a807-74ad-461e-8b15-d609ab421816","Type":"ContainerDied","Data":"a2d6d5de45d712835eaf1878732c6d85db0439318607bc6d7d581e44fdceb5d8"}
Sep 16 18:18:51.061307 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.061278    2560 scope.go:117] "RemoveContainer" containerID="388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e"
Sep 16 18:18:51.069866 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.069847    2560 scope.go:117] "RemoveContainer" containerID="388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e"
Sep 16 18:18:51.070137 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:18:51.070117    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e\": container with ID starting with 388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e not found: ID does not exist" containerID="388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e"
Sep 16 18:18:51.070207 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.070146    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e"} err="failed to get container status \"388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e\": rpc error: code = NotFound desc = could not find container \"388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e\": container with ID starting with 388a52b4d0e1faf93d49a8cdfeca3d91c39d2c0d0f640e013025e97d83adb60e not found: ID does not exist"
Sep 16 18:18:51.082765 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.082739    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"]
Sep 16 18:18:51.087649 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.087629    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche8g8ht"]
Sep 16 18:18:51.117337 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:51.117316    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6326a807-74ad-461e-8b15-d609ab421816" path="/var/lib/kubelet/pods/6326a807-74ad-461e-8b15-d609ab421816/volumes"
Sep 16 18:18:57.872158 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872118    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"]
Sep 16 18:18:57.872653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872412    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main"
Sep 16 18:18:57.872653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872423    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main"
Sep 16 18:18:57.872653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872444    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" containerName="main"
Sep 16 18:18:57.872653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872449    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" containerName="main"
Sep 16 18:18:57.872653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872492    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0fe281-7fe8-45a7-9f1a-e2c26fb98642" containerName="main"
Sep 16 18:18:57.872653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.872499    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6326a807-74ad-461e-8b15-d609ab421816" containerName="main"
Sep 16 18:18:57.875575 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.875552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:57.879538 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.879517    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 18:18:57.879672 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.879545    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 18:18:57.879829 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.879806    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:18:57.879900 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.879883    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 18:18:57.879956 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.879893    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-tzmvn\""
Sep 16 18:18:57.880321 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.880306    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Sep 16 18:18:57.899454 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:57.899430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"]
Sep 16 18:18:58.043306 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5bbfae5-88d4-4214-9431-5b5c0b96579a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043306 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043337    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043354    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgw8\" (UniqueName: \"kubernetes.io/projected/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kube-api-access-hcgw8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043510    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.043792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.043629    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.143768 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.143688    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"]
Sep 16 18:18:58.144379 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144448 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgw8\" (UniqueName: \"kubernetes.io/projected/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kube-api-access-hcgw8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144448 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144426    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144448 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144460    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5bbfae5-88d4-4214-9431-5b5c0b96579a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144612 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144540    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144855 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144969 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.144969 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.144907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.145058 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.145034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.145542 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.145513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5bbfae5-88d4-4214-9431-5b5c0b96579a-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.146914 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.146887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.147129 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.147116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.147209 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.147191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.149090 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.149074    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-264cv\""
Sep 16 18:18:58.153327 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.153310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgw8\" (UniqueName: \"kubernetes.io/projected/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kube-api-access-hcgw8\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.159524 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.159504    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"]
Sep 16 18:18:58.186522 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.186503    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:18:58.245553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.245331    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6d8c3b-d397-4076-8324-d19c74bbc44e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.245553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.245421    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4d2x\" (UniqueName: \"kubernetes.io/projected/ba6d8c3b-d397-4076-8324-d19c74bbc44e-kube-api-access-l4d2x\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.346452 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.346419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6d8c3b-d397-4076-8324-d19c74bbc44e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.346643 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.346474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4d2x\" (UniqueName: \"kubernetes.io/projected/ba6d8c3b-d397-4076-8324-d19c74bbc44e-kube-api-access-l4d2x\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.348937 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.348912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6d8c3b-d397-4076-8324-d19c74bbc44e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.354339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.354313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4d2x\" (UniqueName: \"kubernetes.io/projected/ba6d8c3b-d397-4076-8324-d19c74bbc44e-kube-api-access-l4d2x\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.465532 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.465472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"]
Sep 16 18:18:58.468278 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:18:58.468250    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bbfae5_88d4_4214_9431_5b5c0b96579a.slice/crio-13e9596ddef5fa455d8908107c733a3ba726ed05f8f1896b7dda0e1e8f40260e WatchSource:0}: Error finding container 13e9596ddef5fa455d8908107c733a3ba726ed05f8f1896b7dda0e1e8f40260e: Status 404 returned error can't find the container with id 13e9596ddef5fa455d8908107c733a3ba726ed05f8f1896b7dda0e1e8f40260e
Sep 16 18:18:58.472068 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.472046    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:58.747755 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:58.747730    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"]
Sep 16 18:18:58.750963 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:18:58.750913    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6d8c3b_d397_4076_8324_d19c74bbc44e.slice/crio-b741ed0d83ff3f0e5274ba02d99315418229a3ad6e6ed508c99e62de06ac45c3 WatchSource:0}: Error finding container b741ed0d83ff3f0e5274ba02d99315418229a3ad6e6ed508c99e62de06ac45c3: Status 404 returned error can't find the container with id b741ed0d83ff3f0e5274ba02d99315418229a3ad6e6ed508c99e62de06ac45c3
Sep 16 18:18:59.094665 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:59.094558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" event={"ID":"ba6d8c3b-d397-4076-8324-d19c74bbc44e","Type":"ContainerStarted","Data":"11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad"}
Sep 16 18:18:59.095174 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:59.095138    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:18:59.095300 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:59.095192    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" event={"ID":"ba6d8c3b-d397-4076-8324-d19c74bbc44e","Type":"ContainerStarted","Data":"b741ed0d83ff3f0e5274ba02d99315418229a3ad6e6ed508c99e62de06ac45c3"}
Sep 16 18:18:59.097267 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:59.097161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" event={"ID":"f5bbfae5-88d4-4214-9431-5b5c0b96579a","Type":"ContainerStarted","Data":"18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb"}
Sep 16 18:18:59.097267 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:59.097222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" event={"ID":"f5bbfae5-88d4-4214-9431-5b5c0b96579a","Type":"ContainerStarted","Data":"13e9596ddef5fa455d8908107c733a3ba726ed05f8f1896b7dda0e1e8f40260e"}
Sep 16 18:18:59.114191 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:18:59.114143    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" podStartSLOduration=1.114126108 podStartE2EDuration="1.114126108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:18:59.112358028 +0000 UTC m=+3254.521475276" watchObservedRunningTime="2026-09-16 18:18:59.114126108 +0000 UTC m=+3254.523243356"
Sep 16 18:19:02.110663 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:02.110624    2560 generic.go:356] "Generic (PLEG): container finished" podID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerID="18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb" exitCode=0
Sep 16 18:19:02.110663 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:02.110665    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" event={"ID":"f5bbfae5-88d4-4214-9431-5b5c0b96579a","Type":"ContainerDied","Data":"18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb"}
Sep 16 18:19:03.118689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:03.118647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" event={"ID":"f5bbfae5-88d4-4214-9431-5b5c0b96579a","Type":"ContainerStarted","Data":"b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4"}
Sep 16 18:19:03.144212 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:03.144164    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podStartSLOduration=3.318640303 podStartE2EDuration="6.144146767s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.825506464s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:19:03.143226652 +0000 UTC m=+3258.552343901" watchObservedRunningTime="2026-09-16 18:19:03.144146767 +0000 UTC m=+3258.553264015"
Sep 16 18:19:03.187378 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:03.187348    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:19:08.187229 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:08.187181    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:19:08.188823 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:08.188788    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:19:18.187691 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:18.187652    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:19:28.187265 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:28.187219    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:19:30.102729 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:30.102697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:19:38.187376 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:38.187332    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:19:48.187600 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:48.187555    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:19:58.187081 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:19:58.187040    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:20:08.187269 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:08.187177    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:20:18.197146 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:18.197106    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:20:18.204847 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:18.204817    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:20:42.131761 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:42.131723    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"]
Sep 16 18:20:42.135294 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:42.135267    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"]
Sep 16 18:20:50.103274 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:50.103231    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:20:57.153066 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.153030    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main" containerID="cri-o://11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad" gracePeriod=45
Sep 16 18:20:57.154275 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.154236    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" containerID="cri-o://b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4" gracePeriod=45
Sep 16 18:20:57.198101 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.198055    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 16 18:20:57.790823 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.790800    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:20:57.859936 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.859857    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5bbfae5-88d4-4214-9431-5b5c0b96579a-cabundle-cert\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.859936 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.859903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tls-certs\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.859938    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-dshm\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.859973    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-home\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860003    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kserve-provision-location\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgw8\" (UniqueName: \"kubernetes.io/projected/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kube-api-access-hcgw8\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860093    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-model-cache\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tmp-dir\") pod \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\" (UID: \"f5bbfae5-88d4-4214-9431-5b5c0b96579a\") "
Sep 16 18:20:57.860441 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860286    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bbfae5-88d4-4214-9431-5b5c0b96579a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:20:57.860441 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-home" (OuterVolumeSpecName: "home") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:57.860557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.860497    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:57.862208 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.862181    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-dshm" (OuterVolumeSpecName: "dshm") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:57.862321 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.862303    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kube-api-access-hcgw8" (OuterVolumeSpecName: "kube-api-access-hcgw8") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "kube-api-access-hcgw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:20:57.862429 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.862407    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:20:57.871908 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.871882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:57.882321 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.882297    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5bbfae5-88d4-4214-9431-5b5c0b96579a" (UID: "f5bbfae5-88d4-4214-9431-5b5c0b96579a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:20:57.961003 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.960982    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961003 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961003    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgw8\" (UniqueName: \"kubernetes.io/projected/f5bbfae5-88d4-4214-9431-5b5c0b96579a-kube-api-access-hcgw8\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961012    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961021    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961030    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5bbfae5-88d4-4214-9431-5b5c0b96579a-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961038    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bbfae5-88d4-4214-9431-5b5c0b96579a-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961046    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:57.961135 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:57.961053    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5bbfae5-88d4-4214-9431-5b5c0b96579a-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:20:58.617947 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.617908    2560 generic.go:356] "Generic (PLEG): container finished" podID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerID="b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4" exitCode=0
Sep 16 18:20:58.618363 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.617961    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"
Sep 16 18:20:58.618363 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.618001    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" event={"ID":"f5bbfae5-88d4-4214-9431-5b5c0b96579a","Type":"ContainerDied","Data":"b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4"}
Sep 16 18:20:58.618363 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.618048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj" event={"ID":"f5bbfae5-88d4-4214-9431-5b5c0b96579a","Type":"ContainerDied","Data":"13e9596ddef5fa455d8908107c733a3ba726ed05f8f1896b7dda0e1e8f40260e"}
Sep 16 18:20:58.618363 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.618070    2560 scope.go:117] "RemoveContainer" containerID="b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4"
Sep 16 18:20:58.627264 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.627243    2560 scope.go:117] "RemoveContainer" containerID="18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb"
Sep 16 18:20:58.634871 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.634854    2560 scope.go:117] "RemoveContainer" containerID="b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4"
Sep 16 18:20:58.635120 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:20:58.635102    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4\": container with ID starting with b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4 not found: ID does not exist" containerID="b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4"
Sep 16 18:20:58.635165 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.635128    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4"} err="failed to get container status \"b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4\": rpc error: code = NotFound desc = could not find container \"b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4\": container with ID starting with b78da93fc2435361be63a9fd0097186c750d75ce9c8c0b2d2f7f2e4b3b0043c4 not found: ID does not exist"
Sep 16 18:20:58.635165 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.635147    2560 scope.go:117] "RemoveContainer" containerID="18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb"
Sep 16 18:20:58.635399 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:20:58.635383    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb\": container with ID starting with 18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb not found: ID does not exist" containerID="18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb"
Sep 16 18:20:58.635456 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.635401    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb"} err="failed to get container status \"18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb\": rpc error: code = NotFound desc = could not find container \"18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb\": container with ID starting with 18ca14133892dcf594907851642678fdba79c38b54f92944b0e1460a5f8bc1fb not found: ID does not exist"
Sep 16 18:20:58.641143 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.641117    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"]
Sep 16 18:20:58.646660 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:58.646639    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c559d747b-bx6gj"]
Sep 16 18:20:59.117269 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:20:59.117235    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" path="/var/lib/kubelet/pods/f5bbfae5-88d4-4214-9431-5b5c0b96579a/volumes"
Sep 16 18:21:00.103631 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:00.103578    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:21:10.103431 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:10.103392    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:21:10.103822 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:10.103493    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:21:20.102994 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:20.102953    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:21:27.391266 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.391236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:21:27.485357 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.485333    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4d2x\" (UniqueName: \"kubernetes.io/projected/ba6d8c3b-d397-4076-8324-d19c74bbc44e-kube-api-access-l4d2x\") pod \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") "
Sep 16 18:21:27.485482 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.485400    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6d8c3b-d397-4076-8324-d19c74bbc44e-tls-certs\") pod \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\" (UID: \"ba6d8c3b-d397-4076-8324-d19c74bbc44e\") "
Sep 16 18:21:27.487491 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.487462    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6d8c3b-d397-4076-8324-d19c74bbc44e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba6d8c3b-d397-4076-8324-d19c74bbc44e" (UID: "ba6d8c3b-d397-4076-8324-d19c74bbc44e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:21:27.487491 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.487477    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6d8c3b-d397-4076-8324-d19c74bbc44e-kube-api-access-l4d2x" (OuterVolumeSpecName: "kube-api-access-l4d2x") pod "ba6d8c3b-d397-4076-8324-d19c74bbc44e" (UID: "ba6d8c3b-d397-4076-8324-d19c74bbc44e"). InnerVolumeSpecName "kube-api-access-l4d2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:21:27.586962 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.586934    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4d2x\" (UniqueName: \"kubernetes.io/projected/ba6d8c3b-d397-4076-8324-d19c74bbc44e-kube-api-access-l4d2x\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:21:27.586962 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.586958    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba6d8c3b-d397-4076-8324-d19c74bbc44e-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:21:27.732920 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.732890    2560 generic.go:356] "Generic (PLEG): container finished" podID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerID="11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad" exitCode=0
Sep 16 18:21:27.733097 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.732930    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"
Sep 16 18:21:27.733097 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.732960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" event={"ID":"ba6d8c3b-d397-4076-8324-d19c74bbc44e","Type":"ContainerDied","Data":"11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad"}
Sep 16 18:21:27.733097 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.733014    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b" event={"ID":"ba6d8c3b-d397-4076-8324-d19c74bbc44e","Type":"ContainerDied","Data":"b741ed0d83ff3f0e5274ba02d99315418229a3ad6e6ed508c99e62de06ac45c3"}
Sep 16 18:21:27.733097 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.733043    2560 scope.go:117] "RemoveContainer" containerID="11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad"
Sep 16 18:21:27.742063 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.742044    2560 scope.go:117] "RemoveContainer" containerID="11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad"
Sep 16 18:21:27.742317 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:21:27.742298    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad\": container with ID starting with 11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad not found: ID does not exist" containerID="11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad"
Sep 16 18:21:27.742361 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.742324    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad"} err="failed to get container status \"11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad\": rpc error: code = NotFound desc = could not find container \"11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad\": container with ID starting with 11638cdde63a857c08755bcbbe8779760a1cf984e7d23d59f45a4270d97042ad not found: ID does not exist"
Sep 16 18:21:27.763312 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.763287    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"]
Sep 16 18:21:27.772285 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:27.772265    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-sche2z22b"]
Sep 16 18:21:29.117024 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:29.116990    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" path="/var/lib/kubelet/pods/ba6d8c3b-d397-4076-8324-d19c74bbc44e/volumes"
Sep 16 18:21:32.171519 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171482    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"]
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171772    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171784    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171796    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171802    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171812    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="storage-initializer"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171818    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="storage-initializer"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171873    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6d8c3b-d397-4076-8324-d19c74bbc44e" containerName="main"
Sep 16 18:21:32.172000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.171880    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5bbfae5-88d4-4214-9431-5b5c0b96579a" containerName="main"
Sep 16 18:21:32.174794 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.174773    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.177660 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.177634    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 18:21:32.177801 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.177634    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-z6flc\""
Sep 16 18:21:32.177801 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.177738    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-2jbxr\""
Sep 16 18:21:32.177801 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.177713    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 18:21:32.177801 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.177764    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 18:21:32.184634 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.184612    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"]
Sep 16 18:21:32.329472 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.329435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c939df6-c4ce-4fa9-be32-4f166e446ea9-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.329662 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.329478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lcw\" (UniqueName: \"kubernetes.io/projected/0c939df6-c4ce-4fa9-be32-4f166e446ea9-kube-api-access-k7lcw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.430514 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.430428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c939df6-c4ce-4fa9-be32-4f166e446ea9-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.430514 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.430474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lcw\" (UniqueName: \"kubernetes.io/projected/0c939df6-c4ce-4fa9-be32-4f166e446ea9-kube-api-access-k7lcw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.432979 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.432955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c939df6-c4ce-4fa9-be32-4f166e446ea9-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.450377 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.450346    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lcw\" (UniqueName: \"kubernetes.io/projected/0c939df6-c4ce-4fa9-be32-4f166e446ea9-kube-api-access-k7lcw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.486392 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.486365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:32.762805 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:32.762774    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"]
Sep 16 18:21:32.765918 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:21:32.765885    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c939df6_c4ce_4fa9_be32_4f166e446ea9.slice/crio-119a9f8e901397f46b551900abc7ed01477142c883259c115e4020d06d9b39e9 WatchSource:0}: Error finding container 119a9f8e901397f46b551900abc7ed01477142c883259c115e4020d06d9b39e9: Status 404 returned error can't find the container with id 119a9f8e901397f46b551900abc7ed01477142c883259c115e4020d06d9b39e9
Sep 16 18:21:33.760696 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:33.760659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" event={"ID":"0c939df6-c4ce-4fa9-be32-4f166e446ea9","Type":"ContainerStarted","Data":"f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e"}
Sep 16 18:21:33.761089 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:33.760718    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" event={"ID":"0c939df6-c4ce-4fa9-be32-4f166e446ea9","Type":"ContainerStarted","Data":"119a9f8e901397f46b551900abc7ed01477142c883259c115e4020d06d9b39e9"}
Sep 16 18:21:33.761089 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:33.760814    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:21:33.778022 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:21:33.777975    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" podStartSLOduration=1.777960457 podStartE2EDuration="1.777960457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:21:33.777430286 +0000 UTC m=+3409.186547535" watchObservedRunningTime="2026-09-16 18:21:33.777960457 +0000 UTC m=+3409.187077708"
Sep 16 18:22:04.767449 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:22:04.767411    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:23:46.834009 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:23:46.833977    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"]
Sep 16 18:23:54.767081 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:23:54.767042    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:24:01.852543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:01.852499    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main" containerID="cri-o://f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e" gracePeriod=45
Sep 16 18:24:04.766813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:04.766768    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:24:14.766945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:14.766905    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:24:14.767359 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:14.767010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:24:24.767521 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:24.767472    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:24:32.093001 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.092973    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:24:32.207689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.207656    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c939df6-c4ce-4fa9-be32-4f166e446ea9-tls-certs\") pod \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") "
Sep 16 18:24:32.207841 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.207711    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7lcw\" (UniqueName: \"kubernetes.io/projected/0c939df6-c4ce-4fa9-be32-4f166e446ea9-kube-api-access-k7lcw\") pod \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\" (UID: \"0c939df6-c4ce-4fa9-be32-4f166e446ea9\") "
Sep 16 18:24:32.209993 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.209957    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c939df6-c4ce-4fa9-be32-4f166e446ea9-kube-api-access-k7lcw" (OuterVolumeSpecName: "kube-api-access-k7lcw") pod "0c939df6-c4ce-4fa9-be32-4f166e446ea9" (UID: "0c939df6-c4ce-4fa9-be32-4f166e446ea9"). InnerVolumeSpecName "kube-api-access-k7lcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:24:32.209993 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.209957    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c939df6-c4ce-4fa9-be32-4f166e446ea9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c939df6-c4ce-4fa9-be32-4f166e446ea9" (UID: "0c939df6-c4ce-4fa9-be32-4f166e446ea9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:24:32.309046 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.309007    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c939df6-c4ce-4fa9-be32-4f166e446ea9-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:24:32.309046 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.309040    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7lcw\" (UniqueName: \"kubernetes.io/projected/0c939df6-c4ce-4fa9-be32-4f166e446ea9-kube-api-access-k7lcw\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:24:32.467688 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.467580    2560 generic.go:356] "Generic (PLEG): container finished" podID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerID="f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e" exitCode=0
Sep 16 18:24:32.467688 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.467641    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"
Sep 16 18:24:32.467688 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.467656    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" event={"ID":"0c939df6-c4ce-4fa9-be32-4f166e446ea9","Type":"ContainerDied","Data":"f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e"}
Sep 16 18:24:32.467909 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.467702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh" event={"ID":"0c939df6-c4ce-4fa9-be32-4f166e446ea9","Type":"ContainerDied","Data":"119a9f8e901397f46b551900abc7ed01477142c883259c115e4020d06d9b39e9"}
Sep 16 18:24:32.467909 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.467757    2560 scope.go:117] "RemoveContainer" containerID="f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e"
Sep 16 18:24:32.476533 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.476515    2560 scope.go:117] "RemoveContainer" containerID="f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e"
Sep 16 18:24:32.476829 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:24:32.476807    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e\": container with ID starting with f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e not found: ID does not exist" containerID="f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e"
Sep 16 18:24:32.476905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.476842    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e"} err="failed to get container status \"f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e\": rpc error: code = NotFound desc = could not find container \"f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e\": container with ID starting with f14ecb196c85779a46436aa985ef25d1dd8f131246f1f1249dc0c2cb2d81292e not found: ID does not exist"
Sep 16 18:24:32.488267 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.488243    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"]
Sep 16 18:24:32.492330 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:32.492310    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-schexnfrh"]
Sep 16 18:24:33.117874 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:33.117843    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" path="/var/lib/kubelet/pods/0c939df6-c4ce-4fa9-be32-4f166e446ea9/volumes"
Sep 16 18:24:39.037643 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.037535    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"]
Sep 16 18:24:39.038068 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.037844    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main"
Sep 16 18:24:39.038068 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.037857    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main"
Sep 16 18:24:39.038068 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.037903    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c939df6-c4ce-4fa9-be32-4f166e446ea9" containerName="main"
Sep 16 18:24:39.042260 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.042231    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.045495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.045461    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Sep 16 18:24:39.045495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.045475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Sep 16 18:24:39.045744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.045500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-snngf\""
Sep 16 18:24:39.045744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.045471    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Sep 16 18:24:39.045744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.045470    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-rmztm\""
Sep 16 18:24:39.050244 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.050222    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"]
Sep 16 18:24:39.171764 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.171729    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4195d635-8d1e-4bf0-b2cf-397f70922358-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.171931 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.171772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdlk\" (UniqueName: \"kubernetes.io/projected/4195d635-8d1e-4bf0-b2cf-397f70922358-kube-api-access-lsdlk\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.272717 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.272682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4195d635-8d1e-4bf0-b2cf-397f70922358-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.272886 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.272730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdlk\" (UniqueName: \"kubernetes.io/projected/4195d635-8d1e-4bf0-b2cf-397f70922358-kube-api-access-lsdlk\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.275277 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.275252    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4195d635-8d1e-4bf0-b2cf-397f70922358-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.281617 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.281575    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdlk\" (UniqueName: \"kubernetes.io/projected/4195d635-8d1e-4bf0-b2cf-397f70922358-kube-api-access-lsdlk\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.352236 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.352160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:39.627781 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:39.627696    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"]
Sep 16 18:24:39.632689 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:24:39.632661    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4195d635_8d1e_4bf0_b2cf_397f70922358.slice/crio-39f83806c487bfb67c14460bb79f681d32ed0909e9dee1a58da64133644f9e95 WatchSource:0}: Error finding container 39f83806c487bfb67c14460bb79f681d32ed0909e9dee1a58da64133644f9e95: Status 404 returned error can't find the container with id 39f83806c487bfb67c14460bb79f681d32ed0909e9dee1a58da64133644f9e95
Sep 16 18:24:40.499408 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:40.499371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" event={"ID":"4195d635-8d1e-4bf0-b2cf-397f70922358","Type":"ContainerStarted","Data":"7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1"}
Sep 16 18:24:40.499822 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:40.499427    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" event={"ID":"4195d635-8d1e-4bf0-b2cf-397f70922358","Type":"ContainerStarted","Data":"39f83806c487bfb67c14460bb79f681d32ed0909e9dee1a58da64133644f9e95"}
Sep 16 18:24:40.500000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:40.499982    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:24:40.516571 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:40.516512    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" podStartSLOduration=1.51649215 podStartE2EDuration="1.51649215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:24:40.515388401 +0000 UTC m=+3595.924505650" watchObservedRunningTime="2026-09-16 18:24:40.51649215 +0000 UTC m=+3595.925609402"
Sep 16 18:24:50.859975 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:24:50.859938    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"]
Sep 16 18:25:05.881869 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:05.881805    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main" containerID="cri-o://7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1" gracePeriod=45
Sep 16 18:25:10.863005 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:10.862953    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:25:20.862210 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:20.862163    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:25:30.066697 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.066660    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"]
Sep 16 18:25:30.070495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.070475    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.076883 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.076844    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 18:25:30.077033 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.076901    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:25:30.077033 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.076924    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 16 18:25:30.077033 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.076944    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-l868g\""
Sep 16 18:25:30.077033 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.076855    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 18:25:30.077233 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.077217    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 18:25:30.082763 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.082740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"]
Sep 16 18:25:30.177850 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.177819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-dshm\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178009 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.177856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-kserve-provision-location\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178009 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.177890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-model-cache\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178009 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.177925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkzh\" (UniqueName: \"kubernetes.io/projected/7a71ec40-7aef-4b59-a500-a2098d347cc8-kube-api-access-crkzh\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178009 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.177956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a71ec40-7aef-4b59-a500-a2098d347cc8-tls-certs\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178150 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.178011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-home\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178150 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.178049    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a71ec40-7aef-4b59-a500-a2098d347cc8-cabundle-cert\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.178150 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.178067    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-tmp-dir\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279152 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a71ec40-7aef-4b59-a500-a2098d347cc8-tls-certs\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279152 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-home\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a71ec40-7aef-4b59-a500-a2098d347cc8-cabundle-cert\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279199    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-tmp-dir\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-dshm\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-kserve-provision-location\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279279    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-model-cache\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkzh\" (UniqueName: \"kubernetes.io/projected/7a71ec40-7aef-4b59-a500-a2098d347cc8-kube-api-access-crkzh\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279690 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-home\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279690 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279673    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-kserve-provision-location\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279808 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-model-cache\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279872 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-tmp-dir\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.279988 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.279969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a71ec40-7aef-4b59-a500-a2098d347cc8-cabundle-cert\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.282094 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.282073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-dshm\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.282483 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.282462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a71ec40-7aef-4b59-a500-a2098d347cc8-tls-certs\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.286913 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.286892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkzh\" (UniqueName: \"kubernetes.io/projected/7a71ec40-7aef-4b59-a500-a2098d347cc8-kube-api-access-crkzh\") pod \"stop-feature-test-kserve-57d77dbc48-mw4pd\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.387647 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.387539    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:30.667628 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.667523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"]
Sep 16 18:25:30.670922 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:25:30.670897    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a71ec40_7aef_4b59_a500_a2098d347cc8.slice/crio-a9ce481d247d44c27bf51a387b7a33679a9c4e270a4c9cb61e7a3bbb1605f0e9 WatchSource:0}: Error finding container a9ce481d247d44c27bf51a387b7a33679a9c4e270a4c9cb61e7a3bbb1605f0e9: Status 404 returned error can't find the container with id a9ce481d247d44c27bf51a387b7a33679a9c4e270a4c9cb61e7a3bbb1605f0e9
Sep 16 18:25:30.699793 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.699756    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" event={"ID":"7a71ec40-7aef-4b59-a500-a2098d347cc8","Type":"ContainerStarted","Data":"a9ce481d247d44c27bf51a387b7a33679a9c4e270a4c9cb61e7a3bbb1605f0e9"}
Sep 16 18:25:30.862346 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:30.862302    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:25:31.705445 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:31.705407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" event={"ID":"7a71ec40-7aef-4b59-a500-a2098d347cc8","Type":"ContainerStarted","Data":"b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07"}
Sep 16 18:25:33.715688 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:33.715655    2560 generic.go:356] "Generic (PLEG): container finished" podID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerID="b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07" exitCode=0
Sep 16 18:25:33.716082 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:33.715708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" event={"ID":"7a71ec40-7aef-4b59-a500-a2098d347cc8","Type":"ContainerDied","Data":"b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07"}
Sep 16 18:25:34.722359 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:34.722325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" event={"ID":"7a71ec40-7aef-4b59-a500-a2098d347cc8","Type":"ContainerStarted","Data":"820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0"}
Sep 16 18:25:34.744978 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:34.744926    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podStartSLOduration=2.521201279 podStartE2EDuration="4.744910596s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.223709317s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:25:34.743306989 +0000 UTC m=+3650.152424233" watchObservedRunningTime="2026-09-16 18:25:34.744910596 +0000 UTC m=+3650.154027844"
Sep 16 18:25:35.388096 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:35.388056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:36.130460 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.130437    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:25:36.230750 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.230717    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4195d635-8d1e-4bf0-b2cf-397f70922358-tls-certs\") pod \"4195d635-8d1e-4bf0-b2cf-397f70922358\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") "
Sep 16 18:25:36.230917 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.230759    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsdlk\" (UniqueName: \"kubernetes.io/projected/4195d635-8d1e-4bf0-b2cf-397f70922358-kube-api-access-lsdlk\") pod \"4195d635-8d1e-4bf0-b2cf-397f70922358\" (UID: \"4195d635-8d1e-4bf0-b2cf-397f70922358\") "
Sep 16 18:25:36.233007 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.232975    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4195d635-8d1e-4bf0-b2cf-397f70922358-kube-api-access-lsdlk" (OuterVolumeSpecName: "kube-api-access-lsdlk") pod "4195d635-8d1e-4bf0-b2cf-397f70922358" (UID: "4195d635-8d1e-4bf0-b2cf-397f70922358"). InnerVolumeSpecName "kube-api-access-lsdlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:25:36.233007 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.232996    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4195d635-8d1e-4bf0-b2cf-397f70922358-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4195d635-8d1e-4bf0-b2cf-397f70922358" (UID: "4195d635-8d1e-4bf0-b2cf-397f70922358"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:25:36.332043 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.332004    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4195d635-8d1e-4bf0-b2cf-397f70922358-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:25:36.332043 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.332036    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsdlk\" (UniqueName: \"kubernetes.io/projected/4195d635-8d1e-4bf0-b2cf-397f70922358-kube-api-access-lsdlk\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:25:36.731948 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.731911    2560 generic.go:356] "Generic (PLEG): container finished" podID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerID="7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1" exitCode=0
Sep 16 18:25:36.731948 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.731957    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"
Sep 16 18:25:36.732173 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.731997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" event={"ID":"4195d635-8d1e-4bf0-b2cf-397f70922358","Type":"ContainerDied","Data":"7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1"}
Sep 16 18:25:36.732173 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.732053    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7" event={"ID":"4195d635-8d1e-4bf0-b2cf-397f70922358","Type":"ContainerDied","Data":"39f83806c487bfb67c14460bb79f681d32ed0909e9dee1a58da64133644f9e95"}
Sep 16 18:25:36.732173 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.732076    2560 scope.go:117] "RemoveContainer" containerID="7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1"
Sep 16 18:25:36.740824 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.740802    2560 scope.go:117] "RemoveContainer" containerID="7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1"
Sep 16 18:25:36.741108 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:25:36.741087    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1\": container with ID starting with 7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1 not found: ID does not exist" containerID="7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1"
Sep 16 18:25:36.741171 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.741120    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1"} err="failed to get container status \"7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1\": rpc error: code = NotFound desc = could not find container \"7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1\": container with ID starting with 7e770d38796fda70b869dd13c3808c5f56618d1fdf8a7ff41fc1844a19bf51d1 not found: ID does not exist"
Sep 16 18:25:36.752963 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.752937    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"]
Sep 16 18:25:36.756430 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:36.756409    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-86dbc9cdb7-xcvz7"]
Sep 16 18:25:37.117739 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:37.117663    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" path="/var/lib/kubelet/pods/4195d635-8d1e-4bf0-b2cf-397f70922358/volumes"
Sep 16 18:25:40.387752 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:40.387715    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:25:40.389473 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:40.389442    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:25:50.388867 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:25:50.388816    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:26:00.388223 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:26:00.388175    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:26:10.388000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:26:10.387947    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:26:20.388130 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:26:20.388074    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:26:30.389085 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:26:30.389036    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:26:40.388023 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:26:40.387969    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:26:50.388659 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:26:50.388579    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:27:00.388373 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:00.388326    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:27:10.388905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:10.388856    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:27:20.388733 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:20.388677    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:27:30.398546 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:30.398507    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:27:30.406744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:30.406722    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:27:32.310345 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:32.310312    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"]
Sep 16 18:27:47.334275 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:47.334233    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" containerID="cri-o://820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0" gracePeriod=45
Sep 16 18:27:47.399234 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:47.399190    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Sep 16 18:27:48.078133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.078105    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:27:48.101943 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.101918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a71ec40-7aef-4b59-a500-a2098d347cc8-cabundle-cert\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.101952    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-dshm\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.101989    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-model-cache\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102017    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-kserve-provision-location\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-tmp-dir\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102078 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a71ec40-7aef-4b59-a500-a2098d347cc8-tls-certs\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102323 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102092    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crkzh\" (UniqueName: \"kubernetes.io/projected/7a71ec40-7aef-4b59-a500-a2098d347cc8-kube-api-access-crkzh\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102323 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102123    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-home\") pod \"7a71ec40-7aef-4b59-a500-a2098d347cc8\" (UID: \"7a71ec40-7aef-4b59-a500-a2098d347cc8\") "
Sep 16 18:27:48.102433 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102382    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-model-cache" (OuterVolumeSpecName: "model-cache") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:48.102433 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102313    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a71ec40-7aef-4b59-a500-a2098d347cc8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:27:48.102953 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.102925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-home" (OuterVolumeSpecName: "home") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:48.104549 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.104513    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a71ec40-7aef-4b59-a500-a2098d347cc8-kube-api-access-crkzh" (OuterVolumeSpecName: "kube-api-access-crkzh") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "kube-api-access-crkzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:27:48.104748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.104691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-dshm" (OuterVolumeSpecName: "dshm") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:48.104748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.104691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a71ec40-7aef-4b59-a500-a2098d347cc8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:27:48.116574 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.116546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:48.124201 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.124176    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a71ec40-7aef-4b59-a500-a2098d347cc8" (UID: "7a71ec40-7aef-4b59-a500-a2098d347cc8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:27:48.202899 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202805    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.202899 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202838    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a71ec40-7aef-4b59-a500-a2098d347cc8-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.202899 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202854    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.202899 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202867    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.202899 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202880    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.202899 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202892    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a71ec40-7aef-4b59-a500-a2098d347cc8-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.203179 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202904    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a71ec40-7aef-4b59-a500-a2098d347cc8-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.203179 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.202917    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkzh\" (UniqueName: \"kubernetes.io/projected/7a71ec40-7aef-4b59-a500-a2098d347cc8-kube-api-access-crkzh\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:27:48.280581 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.280550    2560 generic.go:356] "Generic (PLEG): container finished" podID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerID="820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0" exitCode=0
Sep 16 18:27:48.280748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.280635    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"
Sep 16 18:27:48.280748 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.280635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" event={"ID":"7a71ec40-7aef-4b59-a500-a2098d347cc8","Type":"ContainerDied","Data":"820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0"}
Sep 16 18:27:48.280885 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.280762    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd" event={"ID":"7a71ec40-7aef-4b59-a500-a2098d347cc8","Type":"ContainerDied","Data":"a9ce481d247d44c27bf51a387b7a33679a9c4e270a4c9cb61e7a3bbb1605f0e9"}
Sep 16 18:27:48.280885 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.280785    2560 scope.go:117] "RemoveContainer" containerID="820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0"
Sep 16 18:27:48.289495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.289474    2560 scope.go:117] "RemoveContainer" containerID="b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07"
Sep 16 18:27:48.296661 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.296642    2560 scope.go:117] "RemoveContainer" containerID="820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0"
Sep 16 18:27:48.296921 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:27:48.296902    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0\": container with ID starting with 820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0 not found: ID does not exist" containerID="820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0"
Sep 16 18:27:48.296972 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.296931    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0"} err="failed to get container status \"820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0\": rpc error: code = NotFound desc = could not find container \"820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0\": container with ID starting with 820f120d9713c0c03c639b2de5533f2b6213fea2d8838f064f0f00d6c43335b0 not found: ID does not exist"
Sep 16 18:27:48.296972 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.296950    2560 scope.go:117] "RemoveContainer" containerID="b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07"
Sep 16 18:27:48.297226 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:27:48.297206    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07\": container with ID starting with b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07 not found: ID does not exist" containerID="b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07"
Sep 16 18:27:48.297284 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.297230    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07"} err="failed to get container status \"b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07\": rpc error: code = NotFound desc = could not find container \"b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07\": container with ID starting with b16a99cfa9db0297c11ed4b594d4a93f940676325ad28c53b0a582a652cb8f07 not found: ID does not exist"
Sep 16 18:27:48.301625 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.301604    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"]
Sep 16 18:27:48.305996 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:48.305976    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-57d77dbc48-mw4pd"]
Sep 16 18:27:49.122490 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:27:49.118292    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" path="/var/lib/kubelet/pods/7a71ec40-7aef-4b59-a500-a2098d347cc8/volumes"
Sep 16 18:28:01.308871 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.308833    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"]
Sep 16 18:28:01.309206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309113    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main"
Sep 16 18:28:01.309206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309125    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main"
Sep 16 18:28:01.309206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309141    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="storage-initializer"
Sep 16 18:28:01.309206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309147    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="storage-initializer"
Sep 16 18:28:01.309206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309154    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main"
Sep 16 18:28:01.309206 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309161    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main"
Sep 16 18:28:01.309448 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309214    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a71ec40-7aef-4b59-a500-a2098d347cc8" containerName="main"
Sep 16 18:28:01.309448 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.309223    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4195d635-8d1e-4bf0-b2cf-397f70922358" containerName="main"
Sep 16 18:28:01.313865 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.313846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.316384 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.316359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 18:28:01.316519 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.316370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 18:28:01.316519 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.316471    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-qkg6q\""
Sep 16 18:28:01.316519 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.316484    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 18:28:01.316820 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.316781    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-4s8vf\""
Sep 16 18:28:01.320146 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.320124    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"]
Sep 16 18:28:01.405403 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.405368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.405555 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.405479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsm9w\" (UniqueName: \"kubernetes.io/projected/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-kube-api-access-rsm9w\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.506683 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.506647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.506836 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.506706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsm9w\" (UniqueName: \"kubernetes.io/projected/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-kube-api-access-rsm9w\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.509256 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.509228    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.518912 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.518885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsm9w\" (UniqueName: \"kubernetes.io/projected/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-kube-api-access-rsm9w\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.624364 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.624270    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:01.897208 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:01.897114    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"]
Sep 16 18:28:01.900345 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:28:01.900316    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7acff1_8274_425b_8b0b_0f1ed3a27c07.slice/crio-5c36ab871d332de18ba2a80ecfe44d4cdf28d6283eb9bd9fbab2a18036cc167a WatchSource:0}: Error finding container 5c36ab871d332de18ba2a80ecfe44d4cdf28d6283eb9bd9fbab2a18036cc167a: Status 404 returned error can't find the container with id 5c36ab871d332de18ba2a80ecfe44d4cdf28d6283eb9bd9fbab2a18036cc167a
Sep 16 18:28:02.341293 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:02.341252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" event={"ID":"4f7acff1-8274-425b-8b0b-0f1ed3a27c07","Type":"ContainerStarted","Data":"e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038"}
Sep 16 18:28:02.341751 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:02.341331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" event={"ID":"4f7acff1-8274-425b-8b0b-0f1ed3a27c07","Type":"ContainerStarted","Data":"5c36ab871d332de18ba2a80ecfe44d4cdf28d6283eb9bd9fbab2a18036cc167a"}
Sep 16 18:28:02.341751 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:02.341388    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:28:02.359231 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:02.359190    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podStartSLOduration=1.3591770699999999 podStartE2EDuration="1.35917707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:28:02.357286543 +0000 UTC m=+3797.766403801" watchObservedRunningTime="2026-09-16 18:28:02.35917707 +0000 UTC m=+3797.768294317"
Sep 16 18:28:33.347641 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:28:33.347581    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:30:29.334929 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:30:29.334894    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"]
Sep 16 18:30:33.347659 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:30:33.347619    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:30:43.347192 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:30:43.347145    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:30:44.354194 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:30:44.354155    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main" containerID="cri-o://e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038" gracePeriod=45
Sep 16 18:30:53.347272 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:30:53.347236    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:30:53.347698 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:30:53.347358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:31:03.347138 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:03.347098    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:31:13.347744 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:13.347706    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:31:14.597096 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.597072    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:31:14.703933 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.703855    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-tls-certs\") pod \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") "
Sep 16 18:31:14.703933 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.703926    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsm9w\" (UniqueName: \"kubernetes.io/projected/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-kube-api-access-rsm9w\") pod \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\" (UID: \"4f7acff1-8274-425b-8b0b-0f1ed3a27c07\") "
Sep 16 18:31:14.706141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.706120    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f7acff1-8274-425b-8b0b-0f1ed3a27c07" (UID: "4f7acff1-8274-425b-8b0b-0f1ed3a27c07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:31:14.706266 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.706243    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-kube-api-access-rsm9w" (OuterVolumeSpecName: "kube-api-access-rsm9w") pod "4f7acff1-8274-425b-8b0b-0f1ed3a27c07" (UID: "4f7acff1-8274-425b-8b0b-0f1ed3a27c07"). InnerVolumeSpecName "kube-api-access-rsm9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:31:14.804881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.804846    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsm9w\" (UniqueName: \"kubernetes.io/projected/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-kube-api-access-rsm9w\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:31:14.804881 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:14.804878    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f7acff1-8274-425b-8b0b-0f1ed3a27c07-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:31:15.093315 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.093278    2560 generic.go:356] "Generic (PLEG): container finished" podID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerID="e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038" exitCode=0
Sep 16 18:31:15.093495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.093327    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"
Sep 16 18:31:15.093495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.093353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" event={"ID":"4f7acff1-8274-425b-8b0b-0f1ed3a27c07","Type":"ContainerDied","Data":"e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038"}
Sep 16 18:31:15.093495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.093402    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn" event={"ID":"4f7acff1-8274-425b-8b0b-0f1ed3a27c07","Type":"ContainerDied","Data":"5c36ab871d332de18ba2a80ecfe44d4cdf28d6283eb9bd9fbab2a18036cc167a"}
Sep 16 18:31:15.093495 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.093432    2560 scope.go:117] "RemoveContainer" containerID="e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038"
Sep 16 18:31:15.102279 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.102256    2560 scope.go:117] "RemoveContainer" containerID="e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038"
Sep 16 18:31:15.102547 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:31:15.102526    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038\": container with ID starting with e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038 not found: ID does not exist" containerID="e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038"
Sep 16 18:31:15.102653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.102555    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038"} err="failed to get container status \"e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038\": rpc error: code = NotFound desc = could not find container \"e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038\": container with ID starting with e966cdb0ea7975e9835f81e13f9624746efa115ac2880eb193d40b4f1333b038 not found: ID does not exist"
Sep 16 18:31:15.117008 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.116983    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"]
Sep 16 18:31:15.121013 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:15.120992    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-6v9zn"]
Sep 16 18:31:17.117411 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:17.117374    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" path="/var/lib/kubelet/pods/4f7acff1-8274-425b-8b0b-0f1ed3a27c07/volumes"
Sep 16 18:31:43.909423 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.909387    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"]
Sep 16 18:31:43.909873 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.909698    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main"
Sep 16 18:31:43.909873 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.909711    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main"
Sep 16 18:31:43.909873 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.909758    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f7acff1-8274-425b-8b0b-0f1ed3a27c07" containerName="main"
Sep 16 18:31:43.912645 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.912627    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:43.916943 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:31:43.916908    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-62876e81\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:43.916943 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:31:43.916931    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"seaweedfs-s3-creds\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-62876e81\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\"" type="*v1.Secret"
Sep 16 18:31:43.917140 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:31:43.916960    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"odh-kserve-custom-ca-bundle\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-62876e81\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\"" type="*v1.ConfigMap"
Sep 16 18:31:43.917140 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.917057    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 18:31:43.917140 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:31:43.917100    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-zrgww\" is forbidden: User \"system:node:ip-10-0-128-156.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-test-llm-inference-service-62876e81\": no relationship found between node 'ip-10-0-128-156.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrgww\"" type="*v1.Secret"
Sep 16 18:31:43.917268 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.917144    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 18:31:43.919778 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.919759    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 18:31:43.943879 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:43.943853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"]
Sep 16 18:31:44.018068 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgk5\" (UniqueName: \"kubernetes.io/projected/b3893a03-63d9-4982-89c0-6a2296eb8504-kube-api-access-6rgk5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3893a03-63d9-4982-89c0-6a2296eb8504-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018148    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018403 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.018403 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.018281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3893a03-63d9-4982-89c0-6a2296eb8504-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119395 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119357    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119395 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgk5\" (UniqueName: \"kubernetes.io/projected/b3893a03-63d9-4982-89c0-6a2296eb8504-kube-api-access-6rgk5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119664 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3893a03-63d9-4982-89c0-6a2296eb8504-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119664 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119664 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119664 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119664 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3893a03-63d9-4982-89c0-6a2296eb8504-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119926 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119926 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119800    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.119926 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.120046 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.119928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.120086 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.120041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.121830 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.121805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.122069 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.122052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3893a03-63d9-4982-89c0-6a2296eb8504-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.127068 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.127050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgk5\" (UniqueName: \"kubernetes.io/projected/b3893a03-63d9-4982-89c0-6a2296eb8504-kube-api-access-6rgk5\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.854552 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.854516    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 18:31:44.907021 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.906990    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:31:44.910458 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.910434    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3893a03-63d9-4982-89c0-6a2296eb8504-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:44.917802 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:44.917783    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrgww\""
Sep 16 18:31:45.126173 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:45.126097    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrgww\""
Sep 16 18:31:45.133027 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:45.133006    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:45.417159 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:45.417056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"]
Sep 16 18:31:45.420341 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:31:45.420309    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3893a03_63d9_4982_89c0_6a2296eb8504.slice/crio-dcdb2b1db2e54f3cbba59fc0d1b8456f590e66bf2a1e5f25100173eb81cd648d WatchSource:0}: Error finding container dcdb2b1db2e54f3cbba59fc0d1b8456f590e66bf2a1e5f25100173eb81cd648d: Status 404 returned error can't find the container with id dcdb2b1db2e54f3cbba59fc0d1b8456f590e66bf2a1e5f25100173eb81cd648d
Sep 16 18:31:45.424780 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:45.424759    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 16 18:31:46.235952 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:46.235886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" event={"ID":"b3893a03-63d9-4982-89c0-6a2296eb8504","Type":"ContainerStarted","Data":"7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562"}
Sep 16 18:31:46.236464 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:46.235964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" event={"ID":"b3893a03-63d9-4982-89c0-6a2296eb8504","Type":"ContainerStarted","Data":"dcdb2b1db2e54f3cbba59fc0d1b8456f590e66bf2a1e5f25100173eb81cd648d"}
Sep 16 18:31:48.247923 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:48.247884    2560 generic.go:356] "Generic (PLEG): container finished" podID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerID="7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562" exitCode=0
Sep 16 18:31:48.248317 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:48.247957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" event={"ID":"b3893a03-63d9-4982-89c0-6a2296eb8504","Type":"ContainerDied","Data":"7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562"}
Sep 16 18:31:49.254658 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:49.254620    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" event={"ID":"b3893a03-63d9-4982-89c0-6a2296eb8504","Type":"ContainerStarted","Data":"a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92"}
Sep 16 18:31:49.277462 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:49.277334    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podStartSLOduration=3.654658682 podStartE2EDuration="6.277314805s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.622656123s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:31:49.275957158 +0000 UTC m=+4024.685074405" watchObservedRunningTime="2026-09-16 18:31:49.277314805 +0000 UTC m=+4024.686432077"
Sep 16 18:31:55.133635 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:55.133578    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:55.134064 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:55.133659    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:31:55.135028 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:31:55.134998    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:32:05.133562 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:32:05.133523    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:32:15.133844 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:32:15.133802    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:32:25.134406 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:32:25.134361    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:32:35.134509 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:32:35.134467    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:32:45.134224 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:32:45.134168    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:32:55.133451 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:32:55.133409    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:33:05.133636 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:33:05.133570    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:33:15.134152 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:33:15.134109    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:33:25.134368 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:33:25.134320    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:33:35.133817 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:33:35.133769    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Sep 16 18:33:45.143461 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:33:45.143381    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:33:45.151165 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:33:45.151144    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:34:14.636209 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:14.636170    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"]
Sep 16 18:34:29.658422 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:29.658374    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main" containerID="cri-o://a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92" gracePeriod=45
Sep 16 18:34:30.318141 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.318114    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:34:30.399795 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399703    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rgk5\" (UniqueName: \"kubernetes.io/projected/b3893a03-63d9-4982-89c0-6a2296eb8504-kube-api-access-6rgk5\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.399795 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399747    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-home\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.399795 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399779    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3893a03-63d9-4982-89c0-6a2296eb8504-tls-certs\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.400074 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399812    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-model-cache\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.400074 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399833    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3893a03-63d9-4982-89c0-6a2296eb8504-cabundle-cert\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.400074 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399893    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-tmp-dir\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.400074 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399926    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-kserve-provision-location\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.400074 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.399961    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-dshm\") pod \"b3893a03-63d9-4982-89c0-6a2296eb8504\" (UID: \"b3893a03-63d9-4982-89c0-6a2296eb8504\") "
Sep 16 18:34:30.400353 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.400081    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.400353 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.400270    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.400470 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.400382    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3893a03-63d9-4982-89c0-6a2296eb8504-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:34:30.400634 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.400575    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-home" (OuterVolumeSpecName: "home") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.402707 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.402684    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-dshm" (OuterVolumeSpecName: "dshm") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.402797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.402698    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3893a03-63d9-4982-89c0-6a2296eb8504-kube-api-access-6rgk5" (OuterVolumeSpecName: "kube-api-access-6rgk5") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "kube-api-access-6rgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:34:30.402797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.402712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3893a03-63d9-4982-89c0-6a2296eb8504-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:34:30.418342 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.418310    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.422839 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.422811    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3893a03-63d9-4982-89c0-6a2296eb8504" (UID: "b3893a03-63d9-4982-89c0-6a2296eb8504"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:34:30.501260 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501219    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3893a03-63d9-4982-89c0-6a2296eb8504-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.501260 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501252    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.501260 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501261    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.501516 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501270    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.501516 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501285    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rgk5\" (UniqueName: \"kubernetes.io/projected/b3893a03-63d9-4982-89c0-6a2296eb8504-kube-api-access-6rgk5\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.501516 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501296    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3893a03-63d9-4982-89c0-6a2296eb8504-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.501516 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.501304    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3893a03-63d9-4982-89c0-6a2296eb8504-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:34:30.913626 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.913568    2560 generic.go:356] "Generic (PLEG): container finished" podID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerID="a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92" exitCode=0
Sep 16 18:34:30.914101 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.913644    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"
Sep 16 18:34:30.914101 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.913662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" event={"ID":"b3893a03-63d9-4982-89c0-6a2296eb8504","Type":"ContainerDied","Data":"a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92"}
Sep 16 18:34:30.914101 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.913735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm" event={"ID":"b3893a03-63d9-4982-89c0-6a2296eb8504","Type":"ContainerDied","Data":"dcdb2b1db2e54f3cbba59fc0d1b8456f590e66bf2a1e5f25100173eb81cd648d"}
Sep 16 18:34:30.914101 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.913771    2560 scope.go:117] "RemoveContainer" containerID="a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92"
Sep 16 18:34:30.922691 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.922672    2560 scope.go:117] "RemoveContainer" containerID="7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562"
Sep 16 18:34:30.930162 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.930147    2560 scope.go:117] "RemoveContainer" containerID="a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92"
Sep 16 18:34:30.930419 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:34:30.930400    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92\": container with ID starting with a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92 not found: ID does not exist" containerID="a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92"
Sep 16 18:34:30.930471 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.930428    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92"} err="failed to get container status \"a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92\": rpc error: code = NotFound desc = could not find container \"a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92\": container with ID starting with a769f7f783b1fa557c7375842058b30b553c7e11f0d8ce602d839b2bd6ce6f92 not found: ID does not exist"
Sep 16 18:34:30.930471 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.930444    2560 scope.go:117] "RemoveContainer" containerID="7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562"
Sep 16 18:34:30.930734 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:34:30.930711    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562\": container with ID starting with 7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562 not found: ID does not exist" containerID="7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562"
Sep 16 18:34:30.930791 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.930742    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562"} err="failed to get container status \"7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562\": rpc error: code = NotFound desc = could not find container \"7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562\": container with ID starting with 7a29e0c8245590159f0ddfb8d14420994546efe5fedb96ec28081e441f535562 not found: ID does not exist"
Sep 16 18:34:30.950065 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.950039    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"]
Sep 16 18:34:30.956896 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:30.956875    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-bdk59lm"]
Sep 16 18:34:31.117743 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:34:31.117709    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" path="/var/lib/kubelet/pods/b3893a03-63d9-4982-89c0-6a2296eb8504/volumes"
Sep 16 18:35:08.031696 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.031603    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"]
Sep 16 18:35:08.032256 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.032050    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main"
Sep 16 18:35:08.032256 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.032072    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main"
Sep 16 18:35:08.032256 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.032095    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="storage-initializer"
Sep 16 18:35:08.032256 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.032104    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="storage-initializer"
Sep 16 18:35:08.032256 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.032190    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3893a03-63d9-4982-89c0-6a2296eb8504" containerName="main"
Sep 16 18:35:08.034413 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.034391    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.042263 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.042242    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 16 18:35:08.042385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.042260    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 18:35:08.042385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.042269    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:35:08.042803 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.042784    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 18:35:08.042904 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.042786    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 18:35:08.043465 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.043439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-6x9cs\""
Sep 16 18:35:08.043573 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.043503    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-dnj2m\""
Sep 16 18:35:08.060238 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.060200    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"]
Sep 16 18:35:08.108833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.108798    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.108833 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.108831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-dshm\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.109063 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.108851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.109063 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.108963    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.109063 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.108994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.109063 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.109032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.109249 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.109093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-home\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.109249 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.109123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw789\" (UniqueName: \"kubernetes.io/projected/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kube-api-access-rw789\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.209885 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.209850    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.209885 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.209883    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-dshm\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.209911    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.209954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.209974    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-home\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210368 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw789\" (UniqueName: \"kubernetes.io/projected/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kube-api-access-rw789\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210368 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210492 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210384    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210492 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210409    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.210620 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.210578    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-home\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.211089 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.211061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.212388 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.212365    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-dshm\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.212573 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.212553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.220139 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.220109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw789\" (UniqueName: \"kubernetes.io/projected/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kube-api-access-rw789\") pod \"custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.343466 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.343402    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:08.622640 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.622545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"]
Sep 16 18:35:08.626252 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:35:08.626222    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7345d0_9855_4d68_80f7_c8c9b1b4308b.slice/crio-3b7a715c0bed4b963150e856812119a10be71d13768bb818d779bea532bfc053 WatchSource:0}: Error finding container 3b7a715c0bed4b963150e856812119a10be71d13768bb818d779bea532bfc053: Status 404 returned error can't find the container with id 3b7a715c0bed4b963150e856812119a10be71d13768bb818d779bea532bfc053
Sep 16 18:35:08.628226 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:08.628198    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:35:09.060709 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:09.060667    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerStarted","Data":"3b7a715c0bed4b963150e856812119a10be71d13768bb818d779bea532bfc053"}
Sep 16 18:35:11.071522 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:11.071483    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerStarted","Data":"dceff41c762d1a74228a7937c896d36661b4bc5837d9efa261459c5f3b9346fa"}
Sep 16 18:35:11.071916 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:11.071577    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:12.077825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:12.077789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerStarted","Data":"90dfc3ec0b40de15d54a45e4d086e9ab0c749dde7bfe45ce0162e7c87c0c4447"}
Sep 16 18:35:14.089064 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:14.089022    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerID="90dfc3ec0b40de15d54a45e4d086e9ab0c749dde7bfe45ce0162e7c87c0c4447" exitCode=0
Sep 16 18:35:14.089565 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:14.089099    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerDied","Data":"90dfc3ec0b40de15d54a45e4d086e9ab0c749dde7bfe45ce0162e7c87c0c4447"}
Sep 16 18:35:15.095598 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:15.095550    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerStarted","Data":"70b7945691ccf6f0f7b48384d215679a44d51b691731dbfc49e7c4f41d07b317"}
Sep 16 18:35:15.124575 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:15.124505    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podStartSLOduration=3.32158677 podStartE2EDuration="7.12448577s" totalImagesPullingTime="1.684674224s" totalInitContainerRuntime="2.118224776s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:35:15.123151782 +0000 UTC m=+4230.532269033" watchObservedRunningTime="2026-09-16 18:35:15.12448577 +0000 UTC m=+4230.533603019"
Sep 16 18:35:18.344508 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:18.344471    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:18.345045 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:18.344702    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:18.346122 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:18.346088    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:35:28.344605 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:28.344550    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:35:29.131559 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:29.131529    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:35:38.344330 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:38.344280    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:35:48.344330 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:48.344279    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:35:58.344734 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:35:58.344688    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:08.343944 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:08.343900    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:18.344793 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:18.344739    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:28.343967 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:28.343921    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:38.344917 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:38.344813    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:48.343879 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:48.343818    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:58.344420 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:58.344373    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8001/health\": dial tcp 10.134.0.66:8001: connect: connection refused"
Sep 16 18:36:58.992621 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:58.992576    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"]
Sep 16 18:36:58.998487 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:58.998464    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.002305 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.002275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 18:36:59.002419 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.002285    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 18:36:59.002419 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.002332    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-ztzl2\""
Sep 16 18:36:59.002547 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.002510    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-mbk5l\""
Sep 16 18:36:59.002807 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.002790    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 18:36:59.013606 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.011333    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"]
Sep 16 18:36:59.130439 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.130389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrqqb\" (UniqueName: \"kubernetes.io/projected/9ade6026-772d-4c14-9b30-84491926005c-kube-api-access-jrqqb\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.130745 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.130455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade6026-772d-4c14-9b30-84491926005c-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.231789 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.231737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade6026-772d-4c14-9b30-84491926005c-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.231997 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.231898    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrqqb\" (UniqueName: \"kubernetes.io/projected/9ade6026-772d-4c14-9b30-84491926005c-kube-api-access-jrqqb\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.238634 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.238583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade6026-772d-4c14-9b30-84491926005c-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.240551 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.240531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrqqb\" (UniqueName: \"kubernetes.io/projected/9ade6026-772d-4c14-9b30-84491926005c-kube-api-access-jrqqb\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.317723 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.317640    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:36:59.600491 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:36:59.600398    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"]
Sep 16 18:36:59.618136 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:36:59.618102    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ade6026_772d_4c14_9b30_84491926005c.slice/crio-263771fc2de31078d96926b7b9e2d611905e50436fc35aa195f88204d16036c2 WatchSource:0}: Error finding container 263771fc2de31078d96926b7b9e2d611905e50436fc35aa195f88204d16036c2: Status 404 returned error can't find the container with id 263771fc2de31078d96926b7b9e2d611905e50436fc35aa195f88204d16036c2
Sep 16 18:37:00.534677 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:00.534633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" event={"ID":"9ade6026-772d-4c14-9b30-84491926005c","Type":"ContainerStarted","Data":"728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911"}
Sep 16 18:37:00.534990 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:00.534953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" event={"ID":"9ade6026-772d-4c14-9b30-84491926005c","Type":"ContainerStarted","Data":"263771fc2de31078d96926b7b9e2d611905e50436fc35aa195f88204d16036c2"}
Sep 16 18:37:00.535142 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:00.535008    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:37:00.556162 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:00.556104    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" podStartSLOduration=2.556085393 podStartE2EDuration="2.556085393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:37:00.552885595 +0000 UTC m=+4335.962002852" watchObservedRunningTime="2026-09-16 18:37:00.556085393 +0000 UTC m=+4335.965202655"
Sep 16 18:37:08.359754 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:08.359721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:37:08.371876 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:08.371851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:37:19.901361 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:19.901320    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"]
Sep 16 18:37:31.540815 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:31.540782    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:37:34.924074 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:34.924022    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main" containerID="cri-o://70b7945691ccf6f0f7b48384d215679a44d51b691731dbfc49e7c4f41d07b317" gracePeriod=45
Sep 16 18:37:35.535099 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.535065    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="llm-d-routing-sidecar" containerID="cri-o://dceff41c762d1a74228a7937c896d36661b4bc5837d9efa261459c5f3b9346fa" gracePeriod=45
Sep 16 18:37:35.690025 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.689991    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerID="70b7945691ccf6f0f7b48384d215679a44d51b691731dbfc49e7c4f41d07b317" exitCode=0
Sep 16 18:37:35.690025 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.690019    2560 generic.go:356] "Generic (PLEG): container finished" podID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerID="dceff41c762d1a74228a7937c896d36661b4bc5837d9efa261459c5f3b9346fa" exitCode=0
Sep 16 18:37:35.690227 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.690030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerDied","Data":"70b7945691ccf6f0f7b48384d215679a44d51b691731dbfc49e7c4f41d07b317"}
Sep 16 18:37:35.690227 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.690098    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerDied","Data":"dceff41c762d1a74228a7937c896d36661b4bc5837d9efa261459c5f3b9346fa"}
Sep 16 18:37:35.780996 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.780965    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:37:35.825389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tmp-dir\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825534 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825403    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-home\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825534 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-dshm\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825534 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825465    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-cabundle-cert\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825534 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825496    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw789\" (UniqueName: \"kubernetes.io/projected/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kube-api-access-rw789\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kserve-provision-location\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825566    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tls-certs\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825792 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825609    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-model-cache\") pod \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\" (UID: \"1a7345d0-9855-4d68-80f7-c8c9b1b4308b\") "
Sep 16 18:37:35.825949 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:37:35.826000 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.825981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-home" (OuterVolumeSpecName: "home") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.826111 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.826082    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.828458 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.828427    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:37:35.828553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.828536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-dshm" (OuterVolumeSpecName: "dshm") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.828639 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.828546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kube-api-access-rw789" (OuterVolumeSpecName: "kube-api-access-rw789") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "kube-api-access-rw789". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:37:35.841520 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.841488    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.848554 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.848532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1a7345d0-9855-4d68-80f7-c8c9b1b4308b" (UID: "1a7345d0-9855-4d68-80f7-c8c9b1b4308b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:37:35.926862 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926834    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.926862 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926858    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.926862 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926868    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.927233 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926876    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.927233 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926884    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw789\" (UniqueName: \"kubernetes.io/projected/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kube-api-access-rw789\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.927233 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926893    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.927233 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926903    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:35.927233 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:35.926911    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a7345d0-9855-4d68-80f7-c8c9b1b4308b-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:37:36.695797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.695767    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"
Sep 16 18:37:36.695797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.695775    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z" event={"ID":"1a7345d0-9855-4d68-80f7-c8c9b1b4308b","Type":"ContainerDied","Data":"3b7a715c0bed4b963150e856812119a10be71d13768bb818d779bea532bfc053"}
Sep 16 18:37:36.696045 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.695839    2560 scope.go:117] "RemoveContainer" containerID="70b7945691ccf6f0f7b48384d215679a44d51b691731dbfc49e7c4f41d07b317"
Sep 16 18:37:36.704687 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.704664    2560 scope.go:117] "RemoveContainer" containerID="90dfc3ec0b40de15d54a45e4d086e9ab0c749dde7bfe45ce0162e7c87c0c4447"
Sep 16 18:37:36.712133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.712116    2560 scope.go:117] "RemoveContainer" containerID="dceff41c762d1a74228a7937c896d36661b4bc5837d9efa261459c5f3b9346fa"
Sep 16 18:37:36.717779 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.717758    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"]
Sep 16 18:37:36.723726 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:36.723707    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6669f777b9-k5p4z"]
Sep 16 18:37:37.116852 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:37:37.116822    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" path="/var/lib/kubelet/pods/1a7345d0-9855-4d68-80f7-c8c9b1b4308b/volumes"
Sep 16 18:38:27.045999 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.045960    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"]
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046264    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="llm-d-routing-sidecar"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046279    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="llm-d-routing-sidecar"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046301    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046308    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046315    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="storage-initializer"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046321    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="storage-initializer"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046369    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="main"
Sep 16 18:38:27.046407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.046378    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7345d0-9855-4d68-80f7-c8c9b1b4308b" containerName="llm-d-routing-sidecar"
Sep 16 18:38:27.049402 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.049384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.051813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.051773    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 18:38:27.051970 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.051825    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 18:38:27.051970 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.051828    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:38:27.051970 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.051939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-5dnbh\""
Sep 16 18:38:27.052110 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.052098    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-26jvd\""
Sep 16 18:38:27.052158 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.052130    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 16 18:38:27.052453 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.052435    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 18:38:27.069913 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.069885    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"]
Sep 16 18:38:27.127481 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127444    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-dshm\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127517    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxjxz\" (UniqueName: \"kubernetes.io/projected/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kube-api-access-xxjxz\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127689 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tls-certs\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-model-cache\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127722    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.127858 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.127752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-home\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.228645 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-dshm\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.228845 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxjxz\" (UniqueName: \"kubernetes.io/projected/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kube-api-access-xxjxz\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.228845 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.228845 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tls-certs\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.228845 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-model-cache\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.228936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-home\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.229036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229133 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.229100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-model-cache\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229357 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.229241    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tmp-dir\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229357 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.229314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-home\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229512 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.229486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.229716 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.229693    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.231218 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.231191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-dshm\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.231942 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.231921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tls-certs\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.241949 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.241927    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxjxz\" (UniqueName: \"kubernetes.io/projected/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kube-api-access-xxjxz\") pod \"router-with-refs-pd-test-kserve-db5bd79bf-m7sjj\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.358826 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.358732    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:27.672499 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:38:27.672470    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ffcb72a_6e08_46c8_af0d_1888f4ea5e4f.slice/crio-006dbb11f1d11096d512151704bc28217c9eebbdae2b09dfe79261c19345709b WatchSource:0}: Error finding container 006dbb11f1d11096d512151704bc28217c9eebbdae2b09dfe79261c19345709b: Status 404 returned error can't find the container with id 006dbb11f1d11096d512151704bc28217c9eebbdae2b09dfe79261c19345709b
Sep 16 18:38:27.673072 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.673047    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"]
Sep 16 18:38:27.891740 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.891702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerStarted","Data":"401567769e6dc0aaa9335f8babb10b314a5a6322621bec4c7e5712c8981c92c5"}
Sep 16 18:38:27.891930 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.891754    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerStarted","Data":"006dbb11f1d11096d512151704bc28217c9eebbdae2b09dfe79261c19345709b"}
Sep 16 18:38:27.891930 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:27.891820    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:28.897904 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:28.897869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerStarted","Data":"aeaaf9100cdcc528d358a177b880535a3be671bac40b8945cc438a39198809a7"}
Sep 16 18:38:30.907077 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:30.907038    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerID="aeaaf9100cdcc528d358a177b880535a3be671bac40b8945cc438a39198809a7" exitCode=0
Sep 16 18:38:30.907455 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:30.907113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerDied","Data":"aeaaf9100cdcc528d358a177b880535a3be671bac40b8945cc438a39198809a7"}
Sep 16 18:38:31.913982 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:31.913941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerStarted","Data":"9f06ca562ea88a50ba95e051abbbcd60e62d6a42433c6c80ec0d2c47da3c9880"}
Sep 16 18:38:31.945366 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:31.945310    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podStartSLOduration=2.7869108970000003 podStartE2EDuration="4.945292683s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.158381786s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:38:31.943557023 +0000 UTC m=+4427.352674284" watchObservedRunningTime="2026-09-16 18:38:31.945292683 +0000 UTC m=+4427.354409930"
Sep 16 18:38:37.359543 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:37.359505    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:37.359955 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:37.359573    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:37.360947 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:37.360908    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:38:37.373407 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:37.373387    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:38:47.359422 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:47.359373    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:38:57.359334 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:38:57.359280    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:39:04.515614 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:04.515559    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"]
Sep 16 18:39:07.360109 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:07.360056    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:39:11.540476 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:11.540433    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:39:17.359476 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:17.359424    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:39:19.542059 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:19.541991    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main" containerID="cri-o://728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911" gracePeriod=45
Sep 16 18:39:21.540294 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:21.540251    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:39:27.359963 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:27.359898    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:39:31.541089 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:31.541042    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:39:31.541557 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:31.541201    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:39:37.359460 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:37.359353    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:39:41.540321 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:41.540278    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:39:47.359647 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:47.359571    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:39:49.787471 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.787445    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:39:49.813440 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.813370    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrqqb\" (UniqueName: \"kubernetes.io/projected/9ade6026-772d-4c14-9b30-84491926005c-kube-api-access-jrqqb\") pod \"9ade6026-772d-4c14-9b30-84491926005c\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") "
Sep 16 18:39:49.813440 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.813426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade6026-772d-4c14-9b30-84491926005c-tls-certs\") pod \"9ade6026-772d-4c14-9b30-84491926005c\" (UID: \"9ade6026-772d-4c14-9b30-84491926005c\") "
Sep 16 18:39:49.815566 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.815534    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ade6026-772d-4c14-9b30-84491926005c-kube-api-access-jrqqb" (OuterVolumeSpecName: "kube-api-access-jrqqb") pod "9ade6026-772d-4c14-9b30-84491926005c" (UID: "9ade6026-772d-4c14-9b30-84491926005c"). InnerVolumeSpecName "kube-api-access-jrqqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:39:49.815696 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.815536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ade6026-772d-4c14-9b30-84491926005c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9ade6026-772d-4c14-9b30-84491926005c" (UID: "9ade6026-772d-4c14-9b30-84491926005c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:39:49.914894 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.914859    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrqqb\" (UniqueName: \"kubernetes.io/projected/9ade6026-772d-4c14-9b30-84491926005c-kube-api-access-jrqqb\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:39:49.914894 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:49.914890    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ade6026-772d-4c14-9b30-84491926005c-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:39:50.244571 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.244536    2560 generic.go:356] "Generic (PLEG): container finished" podID="9ade6026-772d-4c14-9b30-84491926005c" containerID="728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911" exitCode=0
Sep 16 18:39:50.244781 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.244622    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"
Sep 16 18:39:50.244781 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.244644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" event={"ID":"9ade6026-772d-4c14-9b30-84491926005c","Type":"ContainerDied","Data":"728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911"}
Sep 16 18:39:50.244781 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.244704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw" event={"ID":"9ade6026-772d-4c14-9b30-84491926005c","Type":"ContainerDied","Data":"263771fc2de31078d96926b7b9e2d611905e50436fc35aa195f88204d16036c2"}
Sep 16 18:39:50.244781 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.244728    2560 scope.go:117] "RemoveContainer" containerID="728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911"
Sep 16 18:39:50.254702 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.254679    2560 scope.go:117] "RemoveContainer" containerID="728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911"
Sep 16 18:39:50.254979 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:39:50.254957    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911\": container with ID starting with 728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911 not found: ID does not exist" containerID="728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911"
Sep 16 18:39:50.255052 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.254991    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911"} err="failed to get container status \"728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911\": rpc error: code = NotFound desc = could not find container \"728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911\": container with ID starting with 728c6b8d1dc3e420f8803a7c7c6bdb5967f2270c5ded949644397e263ecc0911 not found: ID does not exist"
Sep 16 18:39:50.267155 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.267133    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"]
Sep 16 18:39:50.271636 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:50.271614    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-hzgbw"]
Sep 16 18:39:51.117304 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:51.117261    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ade6026-772d-4c14-9b30-84491926005c" path="/var/lib/kubelet/pods/9ade6026-772d-4c14-9b30-84491926005c/volumes"
Sep 16 18:39:57.359515 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:39:57.359466    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:40:07.359651 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:07.359562    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:40:13.012120 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.012084    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"]
Sep 16 18:40:13.012514 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.012429    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main"
Sep 16 18:40:13.012514 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.012441    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main"
Sep 16 18:40:13.012514 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.012504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ade6026-772d-4c14-9b30-84491926005c" containerName="main"
Sep 16 18:40:13.015197 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.015178    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.017703 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.017676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 18:40:13.018252 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.018231    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 18:40:13.018374 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.018249    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-w6b8r\""
Sep 16 18:40:13.018374 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.018234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 18:40:13.018645 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.018624    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-nv4gx\""
Sep 16 18:40:13.026167 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.026141    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"]
Sep 16 18:40:13.114771 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.114729    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722b0e-dc80-4e57-8770-125ee1e30d38-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.114963 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.114898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr2c\" (UniqueName: \"kubernetes.io/projected/04722b0e-dc80-4e57-8770-125ee1e30d38-kube-api-access-5pr2c\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.216178 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.216133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr2c\" (UniqueName: \"kubernetes.io/projected/04722b0e-dc80-4e57-8770-125ee1e30d38-kube-api-access-5pr2c\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.216385 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.216188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722b0e-dc80-4e57-8770-125ee1e30d38-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.219020 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.218991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722b0e-dc80-4e57-8770-125ee1e30d38-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.225100 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.225076    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr2c\" (UniqueName: \"kubernetes.io/projected/04722b0e-dc80-4e57-8770-125ee1e30d38-kube-api-access-5pr2c\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.327805 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.327717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:13.605516 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:13.605428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"]
Sep 16 18:40:13.609872 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:40:13.609840    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04722b0e_dc80_4e57_8770_125ee1e30d38.slice/crio-c5c17045a3bbed57c50b94f41875d3d2a0bddb5e5a236d3e6e67fd9271622bee WatchSource:0}: Error finding container c5c17045a3bbed57c50b94f41875d3d2a0bddb5e5a236d3e6e67fd9271622bee: Status 404 returned error can't find the container with id c5c17045a3bbed57c50b94f41875d3d2a0bddb5e5a236d3e6e67fd9271622bee
Sep 16 18:40:14.349441 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:14.349398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" event={"ID":"04722b0e-dc80-4e57-8770-125ee1e30d38","Type":"ContainerStarted","Data":"e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697"}
Sep 16 18:40:14.349861 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:14.349455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" event={"ID":"04722b0e-dc80-4e57-8770-125ee1e30d38","Type":"ContainerStarted","Data":"c5c17045a3bbed57c50b94f41875d3d2a0bddb5e5a236d3e6e67fd9271622bee"}
Sep 16 18:40:14.350018 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:14.349981    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:14.367900 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:14.367854    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" podStartSLOduration=2.367839329 podStartE2EDuration="2.367839329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:40:14.367166982 +0000 UTC m=+4529.776284232" watchObservedRunningTime="2026-09-16 18:40:14.367839329 +0000 UTC m=+4529.776956576"
Sep 16 18:40:17.359980 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:17.359934    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8001/health\": dial tcp 10.134.0.68:8001: connect: connection refused"
Sep 16 18:40:27.369006 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:27.368966    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:40:27.381081 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:27.381056    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:40:43.759923 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:43.759891    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"]
Sep 16 18:40:46.360084 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:46.360052    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:40:58.782207 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:58.782148    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main" containerID="cri-o://9f06ca562ea88a50ba95e051abbbcd60e62d6a42433c6c80ec0d2c47da3c9880" gracePeriod=45
Sep 16 18:40:59.398710 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.398677    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="llm-d-routing-sidecar" containerID="cri-o://401567769e6dc0aaa9335f8babb10b314a5a6322621bec4c7e5712c8981c92c5" gracePeriod=45
Sep 16 18:40:59.539641 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.539605    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerID="9f06ca562ea88a50ba95e051abbbcd60e62d6a42433c6c80ec0d2c47da3c9880" exitCode=0
Sep 16 18:40:59.539641 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.539638    2560 generic.go:356] "Generic (PLEG): container finished" podID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerID="401567769e6dc0aaa9335f8babb10b314a5a6322621bec4c7e5712c8981c92c5" exitCode=0
Sep 16 18:40:59.539854 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.539686    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerDied","Data":"9f06ca562ea88a50ba95e051abbbcd60e62d6a42433c6c80ec0d2c47da3c9880"}
Sep 16 18:40:59.539854 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.539758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerDied","Data":"401567769e6dc0aaa9335f8babb10b314a5a6322621bec4c7e5712c8981c92c5"}
Sep 16 18:40:59.642773 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.642745    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:40:59.709973 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.709945    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tls-certs\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.709981    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-model-cache\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710016    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxjxz\" (UniqueName: \"kubernetes.io/projected/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kube-api-access-xxjxz\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710070    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tmp-dir\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710112 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710099    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-home\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710338 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-cabundle-cert\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710338 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710174    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-dshm\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710338 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710227    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kserve-provision-location\") pod \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\" (UID: \"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f\") "
Sep 16 18:40:59.710338 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.710607 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710563    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.710799 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:40:59.710930 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.710898    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-home" (OuterVolumeSpecName: "home") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.712455 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.712423    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kube-api-access-xxjxz" (OuterVolumeSpecName: "kube-api-access-xxjxz") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "kube-api-access-xxjxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:40:59.712455 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.712441    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:40:59.712648 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.712498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-dshm" (OuterVolumeSpecName: "dshm") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.723724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.723699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.733816 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.733794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" (UID: "2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:40:59.811029 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811006    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.811029 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811026    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.811346 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811036    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.811346 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811044    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.811346 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811052    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.811346 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811060    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:40:59.811346 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:40:59.811069    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxjxz\" (UniqueName: \"kubernetes.io/projected/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f-kube-api-access-xxjxz\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:41:00.545437 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.545402    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"
Sep 16 18:41:00.545653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.545407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj" event={"ID":"2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f","Type":"ContainerDied","Data":"006dbb11f1d11096d512151704bc28217c9eebbdae2b09dfe79261c19345709b"}
Sep 16 18:41:00.545653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.545553    2560 scope.go:117] "RemoveContainer" containerID="9f06ca562ea88a50ba95e051abbbcd60e62d6a42433c6c80ec0d2c47da3c9880"
Sep 16 18:41:00.554644 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.554621    2560 scope.go:117] "RemoveContainer" containerID="aeaaf9100cdcc528d358a177b880535a3be671bac40b8945cc438a39198809a7"
Sep 16 18:41:00.561987 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.561968    2560 scope.go:117] "RemoveContainer" containerID="401567769e6dc0aaa9335f8babb10b314a5a6322621bec4c7e5712c8981c92c5"
Sep 16 18:41:00.571938 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.571910    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"]
Sep 16 18:41:00.576057 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:00.576027    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-db5bd79bf-m7sjj"]
Sep 16 18:41:01.117674 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:41:01.117637    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" path="/var/lib/kubelet/pods/2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f/volumes"
Sep 16 18:42:15.609710 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.609676    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"]
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.609967    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="storage-initializer"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.609979    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="storage-initializer"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.609988    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.609993    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.610001    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="llm-d-routing-sidecar"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.610007    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="llm-d-routing-sidecar"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.610061    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="main"
Sep 16 18:42:15.610123 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.610069    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ffcb72a-6e08-46c8-af0d-1888f4ea5e4f" containerName="llm-d-routing-sidecar"
Sep 16 18:42:15.612992 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.612974    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.616088 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.616056    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:42:15.616278 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.616256    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 16 18:42:15.627904 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.627882    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"]
Sep 16 18:42:15.682934 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.682905    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-kserve-provision-location\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683060 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.682935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35492a72-3a2e-4eb4-b440-254c9ddf630a-cabundle-cert\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683060 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.682954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqzbq\" (UniqueName: \"kubernetes.io/projected/35492a72-3a2e-4eb4-b440-254c9ddf630a-kube-api-access-fqzbq\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683060 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.682972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-model-cache\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683060 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.683041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35492a72-3a2e-4eb4-b440-254c9ddf630a-tls-certs\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683208 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.683087    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-dshm\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683208 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.683115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-home\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.683208 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.683173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-tmp-dir\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784214 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-home\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-tmp-dir\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-kserve-provision-location\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35492a72-3a2e-4eb4-b440-254c9ddf630a-cabundle-cert\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqzbq\" (UniqueName: \"kubernetes.io/projected/35492a72-3a2e-4eb4-b440-254c9ddf630a-kube-api-access-fqzbq\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-model-cache\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784389 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35492a72-3a2e-4eb4-b440-254c9ddf630a-tls-certs\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-dshm\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784724 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784687    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-home\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784895 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784722    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-tmp-dir\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.784895 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.784815    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-model-cache\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.785063 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.785044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35492a72-3a2e-4eb4-b440-254c9ddf630a-cabundle-cert\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.785178 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.785158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-kserve-provision-location\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.786681 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.786661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-dshm\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.786972 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.786952    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35492a72-3a2e-4eb4-b440-254c9ddf630a-tls-certs\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.796959 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.796939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqzbq\" (UniqueName: \"kubernetes.io/projected/35492a72-3a2e-4eb4-b440-254c9ddf630a-kube-api-access-fqzbq\") pod \"prestop-hook-test-kserve-6fb696fc7-l6jsv\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:15.923064 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:15.922981    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:16.199671 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:16.199609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"]
Sep 16 18:42:16.201817 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:42:16.201790    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35492a72_3a2e_4eb4_b440_254c9ddf630a.slice/crio-e82387c86ae073785675a3cf336890a82f4235778219b585d7f6deaa90fc2951 WatchSource:0}: Error finding container e82387c86ae073785675a3cf336890a82f4235778219b585d7f6deaa90fc2951: Status 404 returned error can't find the container with id e82387c86ae073785675a3cf336890a82f4235778219b585d7f6deaa90fc2951
Sep 16 18:42:16.841553 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:16.841512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" event={"ID":"35492a72-3a2e-4eb4-b440-254c9ddf630a","Type":"ContainerStarted","Data":"0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d"}
Sep 16 18:42:16.842036 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:16.841599    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" event={"ID":"35492a72-3a2e-4eb4-b440-254c9ddf630a","Type":"ContainerStarted","Data":"e82387c86ae073785675a3cf336890a82f4235778219b585d7f6deaa90fc2951"}
Sep 16 18:42:18.859657 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:18.859617    2560 generic.go:356] "Generic (PLEG): container finished" podID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerID="0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d" exitCode=0
Sep 16 18:42:18.860086 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:18.859692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" event={"ID":"35492a72-3a2e-4eb4-b440-254c9ddf630a","Type":"ContainerDied","Data":"0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d"}
Sep 16 18:42:19.868508 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:19.868473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" event={"ID":"35492a72-3a2e-4eb4-b440-254c9ddf630a","Type":"ContainerStarted","Data":"5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e"}
Sep 16 18:42:19.894995 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:19.894942    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" podStartSLOduration=2.757540206 podStartE2EDuration="4.89492613s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.137385924s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:42:19.892713526 +0000 UTC m=+4655.301830774" watchObservedRunningTime="2026-09-16 18:42:19.89492613 +0000 UTC m=+4655.304043378"
Sep 16 18:42:19.923405 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:19.923373    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:25.923906 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:25.923866    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:25.925521 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:25.925488    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 18:42:35.924423 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:35.924379    2560 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": dial tcp 10.134.0.70:8000: connect: connection refused"
Sep 16 18:42:40.236265 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:40.236171    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"]
Sep 16 18:42:40.238377 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:40.238346    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"]
Sep 16 18:42:46.360326 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:46.360285    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:42:55.256602 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:55.256513    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main" containerID="cri-o://e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697" gracePeriod=45
Sep 16 18:42:55.262001 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:55.261947    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="main" containerID="cri-o://5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e" gracePeriod=45
Sep 16 18:42:56.360378 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:56.360329    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:42:57.019528 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.019494    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6fb696fc7-l6jsv_35492a72-3a2e-4eb4-b440-254c9ddf630a/main/0.log"
Sep 16 18:42:57.022327 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.022300    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:57.022478 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.022461    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6fb696fc7-l6jsv_35492a72-3a2e-4eb4-b440-254c9ddf630a/main/0.log"
Sep 16 18:42:57.022863 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.022843    2560 generic.go:356] "Generic (PLEG): container finished" podID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerID="5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e" exitCode=130
Sep 16 18:42:57.022939 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.022911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" event={"ID":"35492a72-3a2e-4eb4-b440-254c9ddf630a","Type":"ContainerDied","Data":"5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e"}
Sep 16 18:42:57.022978 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.022957    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv" event={"ID":"35492a72-3a2e-4eb4-b440-254c9ddf630a","Type":"ContainerDied","Data":"e82387c86ae073785675a3cf336890a82f4235778219b585d7f6deaa90fc2951"}
Sep 16 18:42:57.023013 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.022979    2560 scope.go:117] "RemoveContainer" containerID="5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e"
Sep 16 18:42:57.030798 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.030782    2560 scope.go:117] "RemoveContainer" containerID="0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d"
Sep 16 18:42:57.033896 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.033872    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35492a72-3a2e-4eb4-b440-254c9ddf630a-tls-certs\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.033997 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.033915    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35492a72-3a2e-4eb4-b440-254c9ddf630a-cabundle-cert\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.033997 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.033942    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqzbq\" (UniqueName: \"kubernetes.io/projected/35492a72-3a2e-4eb4-b440-254c9ddf630a-kube-api-access-fqzbq\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.033997 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.033960    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-dshm\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.033997 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.033984    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-tmp-dir\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.034296 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034238    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-home\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.034296 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:57.034296 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034289    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35492a72-3a2e-4eb4-b440-254c9ddf630a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:42:57.034533 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034348    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-kserve-provision-location\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.034533 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034385    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-model-cache\") pod \"35492a72-3a2e-4eb4-b440-254c9ddf630a\" (UID: \"35492a72-3a2e-4eb4-b440-254c9ddf630a\") "
Sep 16 18:42:57.034675 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034566    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-home" (OuterVolumeSpecName: "home") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:57.034675 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034663    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-home\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.034782 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034682    2560 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/35492a72-3a2e-4eb4-b440-254c9ddf630a-cabundle-cert\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.034782 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034696    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-tmp-dir\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.034886 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.034804    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-model-cache" (OuterVolumeSpecName: "model-cache") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:57.036638 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.036601    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35492a72-3a2e-4eb4-b440-254c9ddf630a-kube-api-access-fqzbq" (OuterVolumeSpecName: "kube-api-access-fqzbq") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "kube-api-access-fqzbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:42:57.036638 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.036612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35492a72-3a2e-4eb4-b440-254c9ddf630a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:42:57.036795 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.036681    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-dshm" (OuterVolumeSpecName: "dshm") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:57.040134 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.040113    2560 scope.go:117] "RemoveContainer" containerID="5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e"
Sep 16 18:42:57.040455 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:42:57.040429    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e\": container with ID starting with 5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e not found: ID does not exist" containerID="5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e"
Sep 16 18:42:57.040538 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.040465    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e"} err="failed to get container status \"5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e\": rpc error: code = NotFound desc = could not find container \"5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e\": container with ID starting with 5c73d72dd513c60f2d27cded994d2947a83f75a11b80482b560acac7e553bf3e not found: ID does not exist"
Sep 16 18:42:57.040538 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.040490    2560 scope.go:117] "RemoveContainer" containerID="0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d"
Sep 16 18:42:57.040799 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:42:57.040778    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d\": container with ID starting with 0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d not found: ID does not exist" containerID="0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d"
Sep 16 18:42:57.040855 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.040806    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d"} err="failed to get container status \"0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d\": rpc error: code = NotFound desc = could not find container \"0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d\": container with ID starting with 0b994f475fb3dd9baf5ad70b516ba2b735fbd609da39ad41b907e94d9d3bca0d not found: ID does not exist"
Sep 16 18:42:57.058905 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.058877    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35492a72-3a2e-4eb4-b440-254c9ddf630a" (UID: "35492a72-3a2e-4eb4-b440-254c9ddf630a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:42:57.135083 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.135058    2560 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-kserve-provision-location\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.135187 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.135084    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-model-cache\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.135187 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.135094    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35492a72-3a2e-4eb4-b440-254c9ddf630a-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.135187 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.135104    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqzbq\" (UniqueName: \"kubernetes.io/projected/35492a72-3a2e-4eb4-b440-254c9ddf630a-kube-api-access-fqzbq\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:57.135187 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:57.135112    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35492a72-3a2e-4eb4-b440-254c9ddf630a-dshm\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:42:58.027198 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:58.027165    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"
Sep 16 18:42:58.062393 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:58.062360    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"]
Sep 16 18:42:58.068780 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:58.068753    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6fb696fc7-l6jsv"]
Sep 16 18:42:59.117793 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:42:59.117754    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" path="/var/lib/kubelet/pods/35492a72-3a2e-4eb4-b440-254c9ddf630a/volumes"
Sep 16 18:43:06.360131 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:06.360094    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:43:06.360537 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:06.360199    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:43:16.359504 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:16.359459    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:43:25.508732 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.508705    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:43:25.559884 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.559819    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pr2c\" (UniqueName: \"kubernetes.io/projected/04722b0e-dc80-4e57-8770-125ee1e30d38-kube-api-access-5pr2c\") pod \"04722b0e-dc80-4e57-8770-125ee1e30d38\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") "
Sep 16 18:43:25.559993 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.559887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722b0e-dc80-4e57-8770-125ee1e30d38-tls-certs\") pod \"04722b0e-dc80-4e57-8770-125ee1e30d38\" (UID: \"04722b0e-dc80-4e57-8770-125ee1e30d38\") "
Sep 16 18:43:25.561823 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.561794    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04722b0e-dc80-4e57-8770-125ee1e30d38-kube-api-access-5pr2c" (OuterVolumeSpecName: "kube-api-access-5pr2c") pod "04722b0e-dc80-4e57-8770-125ee1e30d38" (UID: "04722b0e-dc80-4e57-8770-125ee1e30d38"). InnerVolumeSpecName "kube-api-access-5pr2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:43:25.561972 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.561882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04722b0e-dc80-4e57-8770-125ee1e30d38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04722b0e-dc80-4e57-8770-125ee1e30d38" (UID: "04722b0e-dc80-4e57-8770-125ee1e30d38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:43:25.660797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.660769    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04722b0e-dc80-4e57-8770-125ee1e30d38-tls-certs\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:43:25.660797 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:25.660795    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pr2c\" (UniqueName: \"kubernetes.io/projected/04722b0e-dc80-4e57-8770-125ee1e30d38-kube-api-access-5pr2c\") on node \"ip-10-0-128-156.ec2.internal\" DevicePath \"\""
Sep 16 18:43:26.140546 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.140510    2560 generic.go:356] "Generic (PLEG): container finished" podID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerID="e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697" exitCode=0
Sep 16 18:43:26.140850 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.140549    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"
Sep 16 18:43:26.140850 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.140556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" event={"ID":"04722b0e-dc80-4e57-8770-125ee1e30d38","Type":"ContainerDied","Data":"e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697"}
Sep 16 18:43:26.140850 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.140675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft" event={"ID":"04722b0e-dc80-4e57-8770-125ee1e30d38","Type":"ContainerDied","Data":"c5c17045a3bbed57c50b94f41875d3d2a0bddb5e5a236d3e6e67fd9271622bee"}
Sep 16 18:43:26.140850 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.140702    2560 scope.go:117] "RemoveContainer" containerID="e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697"
Sep 16 18:43:26.149532 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.149517    2560 scope.go:117] "RemoveContainer" containerID="e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697"
Sep 16 18:43:26.149853 ip-10-0-128-156 kubenswrapper[2560]: E0916 18:43:26.149823    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697\": container with ID starting with e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697 not found: ID does not exist" containerID="e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697"
Sep 16 18:43:26.149951 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.149861    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697"} err="failed to get container status \"e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697\": rpc error: code = NotFound desc = could not find container \"e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697\": container with ID starting with e77625fbd208f95b37dcd37f86490d94f9096cd1a510cdf57f2c587077f99697 not found: ID does not exist"
Sep 16 18:43:26.163475 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.163452    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"]
Sep 16 18:43:26.165621 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:26.165602    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-tzcft"]
Sep 16 18:43:27.117339 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:27.117304    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" path="/var/lib/kubelet/pods/04722b0e-dc80-4e57-8770-125ee1e30d38/volumes"
Sep 16 18:43:53.528469 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:53.528431    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/0.log"
Sep 16 18:43:53.543052 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:53.543021    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/1.log"
Sep 16 18:43:54.411676 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:54.411636    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/0.log"
Sep 16 18:43:54.427080 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:54.427050    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/1.log"
Sep 16 18:43:55.727741 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:55.727702    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5zj2x_d9ecf4ad-bb99-41dd-bac9-c90a542ddf94/authorino/0.log"
Sep 16 18:43:55.870254 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:43:55.870224    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5d8zg_30f0160f-0edd-4747-bcf8-13d0a18a6ec2/kuadrant-console-plugin/0.log"
Sep 16 18:44:05.085842 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:05.085808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5sjzg_0640143a-e0a6-4db6-85be-93f4cf337db9/global-pull-secret-syncer/0.log"
Sep 16 18:44:05.312534 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:05.312498    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s2f66_254141ee-0a25-484f-b28a-0b1485f4ffc9/konnectivity-agent/0.log"
Sep 16 18:44:05.334761 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:05.334731    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-156.ec2.internal_e1d152f82d086cc4921e20e44fe174b4/haproxy/0.log"
Sep 16 18:44:10.139050 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:10.138968    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d9898bd99-5zj2x_d9ecf4ad-bb99-41dd-bac9-c90a542ddf94/authorino/0.log"
Sep 16 18:44:10.227458 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:10.227428    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5d8zg_30f0160f-0edd-4747-bcf8-13d0a18a6ec2/kuadrant-console-plugin/0.log"
Sep 16 18:44:12.503772 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.503741    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw8hl_ce4ff827-ef40-479c-a127-d67bba6be3e7/node-exporter/0.log"
Sep 16 18:44:12.539492 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.539449    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw8hl_ce4ff827-ef40-479c-a127-d67bba6be3e7/kube-rbac-proxy/0.log"
Sep 16 18:44:12.579185 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.579163    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lw8hl_ce4ff827-ef40-479c-a127-d67bba6be3e7/init-textfile/0.log"
Sep 16 18:44:12.620441 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.620418    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-nfc4b_8a82b3b4-164d-4b02-ad13-a0b6972f33bf/kube-rbac-proxy-main/0.log"
Sep 16 18:44:12.647328 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.647306    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-nfc4b_8a82b3b4-164d-4b02-ad13-a0b6972f33bf/kube-rbac-proxy-self/0.log"
Sep 16 18:44:12.675088 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.675025    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-nfc4b_8a82b3b4-164d-4b02-ad13-a0b6972f33bf/openshift-state-metrics/0.log"
Sep 16 18:44:12.978669 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:12.978642    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-cbpg2_fff31bbe-8f93-4523-b374-b7e691f67780/prometheus-operator-admission-webhook/0.log"
Sep 16 18:44:13.124749 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:13.124718    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898987c6b-cb8wh_b6474a2f-907a-4ff2-bcf9-77f63789d1a1/thanos-query/0.log"
Sep 16 18:44:13.149125 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:13.149100    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898987c6b-cb8wh_b6474a2f-907a-4ff2-bcf9-77f63789d1a1/kube-rbac-proxy-web/0.log"
Sep 16 18:44:13.177945 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:13.177916    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898987c6b-cb8wh_b6474a2f-907a-4ff2-bcf9-77f63789d1a1/kube-rbac-proxy/0.log"
Sep 16 18:44:13.205647 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:13.205626    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898987c6b-cb8wh_b6474a2f-907a-4ff2-bcf9-77f63789d1a1/prom-label-proxy/0.log"
Sep 16 18:44:13.232633 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:13.232539    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898987c6b-cb8wh_b6474a2f-907a-4ff2-bcf9-77f63789d1a1/kube-rbac-proxy-rules/0.log"
Sep 16 18:44:13.256300 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:13.256281    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-898987c6b-cb8wh_b6474a2f-907a-4ff2-bcf9-77f63789d1a1/kube-rbac-proxy-metrics/0.log"
Sep 16 18:44:15.159174 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159141    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"]
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159421    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="storage-initializer"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159433    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="storage-initializer"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159458    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159464    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159471    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="main"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159477    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="main"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159518    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="04722b0e-dc80-4e57-8770-125ee1e30d38" containerName="main"
Sep 16 18:44:15.159535 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.159527    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="35492a72-3a2e-4eb4-b440-254c9ddf630a" containerName="main"
Sep 16 18:44:15.162643 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.162620    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.164914 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.164894    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wzhjv\"/\"kube-root-ca.crt\""
Sep 16 18:44:15.165704 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.165690    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wzhjv\"/\"openshift-service-ca.crt\""
Sep 16 18:44:15.165780 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.165729    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wzhjv\"/\"default-dockercfg-d5znf\""
Sep 16 18:44:15.173447 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.173427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"]
Sep 16 18:44:15.246660 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.246635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grqfh\" (UniqueName: \"kubernetes.io/projected/ac7664d9-8bf9-40ca-a878-fd61565d06b4-kube-api-access-grqfh\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.246825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.246680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-proc\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.246825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.246696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-lib-modules\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.246825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.246713    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-sys\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.246825 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.246795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-podres\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347505 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347474    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-podres\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grqfh\" (UniqueName: \"kubernetes.io/projected/ac7664d9-8bf9-40ca-a878-fd61565d06b4-kube-api-access-grqfh\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347603    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-proc\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-lib-modules\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-sys\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-podres\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347718 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-proc\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347998 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347729    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-sys\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.347998 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.347758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ac7664d9-8bf9-40ca-a878-fd61565d06b4-lib-modules\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.355653 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.355632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grqfh\" (UniqueName: \"kubernetes.io/projected/ac7664d9-8bf9-40ca-a878-fd61565d06b4-kube-api-access-grqfh\") pod \"perf-node-gather-daemonset-8rc99\" (UID: \"ac7664d9-8bf9-40ca-a878-fd61565d06b4\") " pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.410009 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.409942    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bd4bf99f8-clfv8_c9e82abe-34c1-4578-9f02-ef0fcaab51a7/console/0.log"
Sep 16 18:44:15.472695 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.472661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:15.744235 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:15.744169    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"]
Sep 16 18:44:15.747457 ip-10-0-128-156 kubenswrapper[2560]: W0916 18:44:15.747430    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podac7664d9_8bf9_40ca_a878_fd61565d06b4.slice/crio-e4d18993ef9666e6f6dd999021902902a41373cfda661396ecbbe4b645eb63d3 WatchSource:0}: Error finding container e4d18993ef9666e6f6dd999021902902a41373cfda661396ecbbe4b645eb63d3: Status 404 returned error can't find the container with id e4d18993ef9666e6f6dd999021902902a41373cfda661396ecbbe4b645eb63d3
Sep 16 18:44:16.327736 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:16.327698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99" event={"ID":"ac7664d9-8bf9-40ca-a878-fd61565d06b4","Type":"ContainerStarted","Data":"cb65498cbb7f7194f8b59ebc34c002b6f3974b48d1b13f28c520f08e5db898e3"}
Sep 16 18:44:16.328098 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:16.327761    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99" event={"ID":"ac7664d9-8bf9-40ca-a878-fd61565d06b4","Type":"ContainerStarted","Data":"e4d18993ef9666e6f6dd999021902902a41373cfda661396ecbbe4b645eb63d3"}
Sep 16 18:44:16.328098 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:16.327788    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:16.345072 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:16.345031    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99" podStartSLOduration=1.345017678 podStartE2EDuration="1.345017678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:16.344090883 +0000 UTC m=+4771.753208143" watchObservedRunningTime="2026-09-16 18:44:16.345017678 +0000 UTC m=+4771.754134925"
Sep 16 18:44:17.153749 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:17.153715    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xpk25_18dd06e3-0da0-46da-b08e-d45b2396b86b/dns/0.log"
Sep 16 18:44:17.229288 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:17.229265    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xpk25_18dd06e3-0da0-46da-b08e-d45b2396b86b/kube-rbac-proxy/0.log"
Sep 16 18:44:17.489154 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:17.489121    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q426q_3a3f53ed-df0f-4f40-94c1-0fdacaa54f70/dns-node-resolver/0.log"
Sep 16 18:44:18.171804 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:18.171764    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w5vt2_7c76ab09-5936-4425-94f8-dd03433ecfa9/node-ca/0.log"
Sep 16 18:44:19.340229 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:19.340194    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/0.log"
Sep 16 18:44:19.355283 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:19.355255    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/1.log"
Sep 16 18:44:19.945651 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:19.945618    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qlnrn_41b04f03-bcb0-49ad-ac5d-a4cf82768494/serve-healthcheck-canary/0.log"
Sep 16 18:44:20.477735 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:20.477701    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qbdw9_e93d211c-97dd-4665-b91d-f5df364301f3/kube-rbac-proxy/0.log"
Sep 16 18:44:20.504975 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:20.504944    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qbdw9_e93d211c-97dd-4665-b91d-f5df364301f3/exporter/0.log"
Sep 16 18:44:20.527486 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:20.527455    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qbdw9_e93d211c-97dd-4665-b91d-f5df364301f3/extractor/0.log"
Sep 16 18:44:22.341393 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:22.341361    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wzhjv/perf-node-gather-daemonset-8rc99"
Sep 16 18:44:23.832813 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:23.832782    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86b645b5b5-hpbfr_80d5ca89-d28e-49dd-b5c7-8acbaab9b5cd/manager/0.log"
Sep 16 18:44:24.773606 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:24.773566    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/0.log"
Sep 16 18:44:24.793476 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:24.793451    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/1.log"
Sep 16 18:44:26.270466 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:26.270433    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/0.log"
Sep 16 18:44:26.285528 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:26.285502    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-8699b59bbc-nl7lw_1c5bd874-c917-4ebf-994b-7a0954556b05/istio-proxy/1.log"
Sep 16 18:44:33.064906 ip-10-0-128-156 kubenswrapper[2560]: I0916 18:44:33.064876    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5g7vl_55dcaec8-4335-427e-a986-304d6cc2166c/kube-multus/0.log"